Python Workflow

Open-source Python projects categorized as Workflow

Top 23 Python Workflow Projects

  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    Project mention: Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions | dev.to | 2024-02-12

    Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.

  • Prefect

    The easiest way to build, run, and monitor data pipelines at scale.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  • dagster

    An orchestration platform for the development, production, and observation of data assets.

    Project mention: Experience with Dagster.io? | news.ycombinator.com | 2023-07-25
  • PDM

    A modern Python package and dependency manager supporting the latest PEP standards

    Project mention: Implementing Quality Checks In Your Git Workflow With Hooks and pre-commit | dev.to | 2023-12-13

    # See https://pre-commit.com for more information # See https://pre-commit.com/hooks.html for more hooks repos: - repo: https://github.com/pre-commit/pre-commit-hooks rev: v3.2.0 hooks: - id: trailing-whitespace - id: end-of-file-fixer - id: check-yaml - id: check-toml - id: check-added-large-files - repo: local hooks: - id: tox lint name: tox-validation entry: pdm run tox -e test,lint language: system files: ^src\/.+py$|pyproject.toml|^tests\/.+py$ types_or: [python, toml] pass_filenames: false - id: tox docs name: tox-docs language: system entry: pdm run tox -e docs types_or: [python, rst, toml] files: ^src\/.+py$|pyproject.toml|^docs\/ pass_filenames: false - repo: https://github.com/pdm-project/pdm rev: 2.10.4 # a PDM release exposing the hook hooks: - id: pdm-lock-check - repo: https://github.com/jumanjihouse/pre-commit-hooks rev: 3.0.0 hooks: - id: markdownlint

  • taipy

    Turns Data and AI algorithms into production-ready web applications in no time.

    Project mention: Show HN: Building data and AI apps, an alternative to Streamlit | news.ycombinator.com | 2024-02-12
  • zenml

    ZenML 🙏: Build portable, production-ready MLOps pipelines. https://zenml.io.

    Project mention: What are some open-source ML pipeline managers that are easy to use? | /r/mlops | 2023-05-03
  • polyaxon

    MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle

  • Onboard AI

    ChatGPT with full context of any GitHub repo. Onboard AI learns any GitHub repo in minutes and lets you chat with it to locate functionality, understand different parts, and generate new code. Use it for free at app.getonboardai.com.

  • ploomber

    The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

    Project mention: Show HN: JupySQL – a SQL client for Jupyter (ipython-SQL successor) | news.ycombinator.com | 2023-12-06

    - One-click sharing powered by Ploomber Cloud: https://ploomber.io

    Documentation: https://jupysql.ploomber.io

    Note that JupySQL is a fork of ipython-sql; which is no longer actively developed. Catherine, ipython-sql's creator, was kind enough to pass the project to us (check out ipython-sql's README).

    We'd love to learn what you think and what features we can ship for JupySQL to be the best SQL client! Please let us know in the comments!

  • octoDNS

    Tools for managing DNS across multiple providers

    Project mention: Show HN: WireHub – easily create and share WireGuard networks | news.ycombinator.com | 2023-11-05
  • viewflow

    Reusable workflow library for Django

    Project mention: Ask HN: Anyone use a code to mindmap/flowchart tool? | news.ycombinator.com | 2024-02-24

    https://github.com/django-extensions/django-extensions/blob/...

    viewflow supports BPMN: https://github.com/viewflow/viewflow

  • doit

    task management & automation tool

    Project mention: How do you deal with CI, project config, etc. falling out of sync across repos? | /r/ExperiencedDevs | 2023-12-06

    I like mage for Go and doit for Python.

  • airflow-maintenance-dags

    A series of DAGs/Workflows to help maintain the operation of Airflow

    Project mention: Airflow meta-db cleaning | /r/dataengineering | 2023-04-06

    Use these maintenance DAGs from the project https://github.com/teamclairvoyant/airflow-maintenance-dags

  • Spiff

    A powerful workflow engine implemented in pure Python

    Project mention: Siffworkflow 2.0.0 | news.ycombinator.com | 2023-06-16
  • galaxy

    Data intensive science for everyone.

    Project mention: Need for GUIs for bioinformatic tools? | /r/bioinformatics | 2023-06-17

    Maybe it would help you to look at the galaxy project: GitHub main site

  • Sooty

    The SOC Analysts all-in-one CLI tool to automate and speed up workflow.

  • mlrun

    Machine Learning automation and tracking

  • obsei

    Obsei is a low code AI powered automation tool. It can be used in various business flows like social listening, AI based alerting, brand image analysis, comparative study and more .

  • toil

    A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

  • gh-action-pypi-publish

    The blessed :octocat: GitHub Action, for publishing your :package: distribution files to PyPI: https://github.com/marketplace/actions/pypi-publish

    Project mention: PyPI new user and new project registrations temporarily suspended | news.ycombinator.com | 2023-05-20

    > Recently I've seen someone on Reddit trying to automate the creation of PyPI projects through GitHub Actions. The person was complaining that the first deployment couldn't use an API key for that project since it didn't exist. So I'm not surprised some people are trying to do the same for malicious purposes.

    Sorry for the tangent, but: you can do this now! If you use trusted publishing, you can register a "pending publisher" for a project that doesn't exist yet. When the trusted publisher (like GitHub Actions) is used, it'll create the project[1].

    All of this is supported transparently by the official publishing action for GitHub Actions[2].

    [1]: https://docs.pypi.org/trusted-publishers/creating-a-project-...

    [2]: https://github.com/pypa/gh-action-pypi-publish

  • orchestra

    Orchestra is a human-in-the-loop AI system for orchestrating project teams of experts and machines. (by b12io)

  • covalent

    Pythonic tool for running machine-learning/high performance/quantum-computing workflows in heterogeneous environments. (by AgnostiqHQ)

    Project mention: Remote execution of code | /r/Python | 2023-12-05

    Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like

  • alfred-stackexchange

    Search StackOverflow.com from Alfred

  • jug

    Parallel programming with Python

  • WorkOS

    The modern API for authentication & user identity. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020). The latest post mention was on 2024-02-24.

Python Workflow related posts

Index

What are some of the best open-source Workflow projects in Python? This list will help you:

Project Stars
1 Airflow 33,606
2 Prefect 14,114
3 dagster 9,682
4 PDM 6,215
5 taipy 5,824
6 zenml 3,543
7 polyaxon 3,453
8 ploomber 3,335
9 octoDNS 2,920
10 viewflow 2,513
11 doit 1,755
12 airflow-maintenance-dags 1,576
13 Spiff 1,573
14 galaxy 1,279
15 Sooty 1,267
16 mlrun 1,216
17 obsei 1,034
18 toil 865
19 gh-action-pypi-publish 798
20 orchestra 656
21 covalent 630
22 alfred-stackexchange 436
23 jug 408
The modern API for authentication & user identity.
The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
workos.com