Python Workflow

Open-source Python projects categorized as Workflow
Edit details
Topics: Python Data Science Machine Learning Automation HacktoberFest
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Top 23 Python Workflow Projects

  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    • Project mention: Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions | dev.to | 2024-02-12

    Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.

  • Prefect

    The easiest way to build, run, and monitor data pipelines at scale.

    • Project mention: Prefect: A workflow orchestration tool for data pipelines | news.ycombinator.com | 2024-03-13

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • dagster

    An orchestration platform for the development, production, and observation of data assets.

    • Project mention: Experience with Dagster.io? | news.ycombinator.com | 2023-07-25

  • Taipy

    Turns Data and AI algorithms into production-ready web applications in no time.

    • Project mention: Python Day 9: Building Interactive Web Apps without HTML/CSS and JavaScript | dev.to | 2024-04-26

    Taipy is an open-source Python library that enables data scientists and developers to build robust end-to-end data pipelines.

  • PDM

    A modern Python package and dependency manager supporting the latest PEP standards

    • Project mention: Implementing Quality Checks In Your Git Workflow With Hooks and pre-commit | dev.to | 2023-12-13

    # See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-added-large-files - repo: local hooks: - id: tox lint name: tox-validation entry: pdm run tox -e test,lint language: system files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ types_or: [python, toml] pass_filenames: false - id: tox docs name: tox-docs language: system entry: pdm run tox -e docs types_or: [python, rst, toml] files: ^src\/.+py$|pyproject.toml|^docs\/ pass_filenames: false - repo: https://github.com/pdm-project/pdm rev: 2.10.4 # a PDM release exposing the hook hooks: - id: pdm-lock-check - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: - id: markdownlint

  • zenml

    ZenML 🙏: Build portable, production-ready MLOps pipelines. https://zenml.io.

    • Project mention: FLaNK AI - 01 April 2024 | dev.to | 2024-04-01

  • polyaxon

    MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • ploomber

    The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

    • Project mention: Show HN: JupySQL – a SQL client for Jupyter (ipython-SQL successor) | news.ycombinator.com | 2023-12-06

    - One-click sharing powered by Ploomber Cloud: https://ploomber.io

    Documentation: https://jupysql.ploomber.io

    Note that JupySQL is a fork of ipython-sql; which is no longer actively developed. Catherine, ipython-sql's creator, was kind enough to pass the project to us (check out ipython-sql's README).

    We'd love to learn what you think and what features we can ship for JupySQL to be the best SQL client! Please let us know in the comments!

  • octoDNS

    Tools for managing DNS across multiple providers

    • Project mention: Show HN: WireHub – easily create and share WireGuard networks | news.ycombinator.com | 2023-11-05

  • viewflow

    Reusable workflow library for Django

    • Project mention: Ask HN: Anyone use a code to mindmap/flowchart tool? | news.ycombinator.com | 2024-02-24

    https://github.com/django-extensions/django-extensions/blob/...

    viewflow supports BPMN: https://github.com/viewflow/viewflow

  • doit

    task management & automation tool

    • Project mention: How do you deal with CI, project config, etc. falling out of sync across repos? | /r/ExperiencedDevs | 2023-12-06

    I like mage for Go and doit for Python.

  • airflow-maintenance-dags

    A series of DAGs/Workflows to help maintain the operation of Airflow

  • Spiff

    A powerful workflow engine implemented in pure Python

    • Project mention: Siffworkflow 2.0.0 | news.ycombinator.com | 2023-06-16

  • galaxy

    Data intensive science for everyone.

    • Project mention: Need for GUIs for bioinformatic tools? | /r/bioinformatics | 2023-06-17

    Maybe it would help you to look at the galaxy project: GitHub main site

  • mlrun

    MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

  • Sooty

    The SOC Analysts all-in-one CLI tool to automate and speed up workflow.

  • obsei

    Obsei is a low code AI powered automation tool. It can be used in various business flows like social listening, AI based alerting, brand image analysis, comparative study and more .

  • toil

    A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

    • Project mention: Show HN: Hatchet – Open-source distributed task queue | news.ycombinator.com | 2024-03-08

    a little late now, but I wonder if https://github.com/DataBiosphere/toil might meet your requirements

  • gh-action-pypi-publish

    The blessed :octocat: GitHub Action, for publishing your :package: distribution files to PyPI: https://github.com/marketplace/actions/pypi-publish

    • Project mention: PyPI new user and new project registrations temporarily suspended | news.ycombinator.com | 2023-05-20

    > Recently I've seen someone on Reddit trying to automate the creation of PyPI projects through GitHub Actions. The person was complaining that the first deployment couldn't use an API key for that project since it didn't exist. So I'm not surprised some people are trying to do the same for malicious purposes.

    Sorry for the tangent, but: you can do this now! If you use trusted publishing, you can register a "pending publisher" for a project that doesn't exist yet. When the trusted publisher (like GitHub Actions) is used, it'll create the project[1].

    All of this is supported transparently by the official publishing action for GitHub Actions[2].

    [1]: https://docs.pypi.org/trusted-publishers/creating-a-project-...

    [2]: https://github.com/pypa/gh-action-pypi-publish

  • covalent

    Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)

    • Project mention: Remote execution of code | /r/Python | 2023-12-05

    Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like

  • orchestra

    Orchestra is a human-in-the-loop AI system for orchestrating project teams of experts and machines. (by b12io)

  • astronomer-cosmos

    Run your dbt Core projects as Apache Airflow DAGs and Task Groups with a few lines of code

  • alfred-stackexchange

    Search StackOverflow.com from Alfred

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python Workflow related posts

Index

What are some of the best open-source Workflow projects in Python? This list will help you:

Project Stars
1 Airflow 34,627
2 Prefect 14,724
3 dagster 10,274
4 Taipy 8,731
5 PDM 6,616
6 zenml 3,682
7 polyaxon 3,486
8 ploomber 3,387
9 octoDNS 2,986
10 viewflow 2,562
11 doit 1,787
12 airflow-maintenance-dags 1,600
13 Spiff 1,602
14 galaxy 1,315
15 mlrun 1,308
16 Sooty 1,284
17 obsei 1,083
18 toil 872
19 gh-action-pypi-publish 844
20 covalent 698
21 orchestra 659
22 astronomer-cosmos 461
23 alfred-stackexchange 435

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com