Top 23 Python Pipeline Projects

• jina

  14 18,477 9.5 Python

  🔮 Build multimodal AI services via cloud native technologies

  

  Project mention: Cross data type search that wasn’t supported well using Elasticsearch | /r/learnprogramming | 2023-04-11

  Jina mainly because of their use of neural networks and AI.

• Prefect

  4 12,078 10.0 Python

  The easiest way to build, run, and monitor data pipelines at scale.

  

  Project mention: self hosted Alternative to easycron.com? | /r/selfhosted | 2022-12-30

• Sonar

  www.sonarsource.com

  sponsored

  Write Clean Python Code. Always.. Sonar helps you commit clean code every time. With over 225 unique rules to find Python bugs, code smells & vulnerabilities, Sonar finds the issues while you focus on the work.

  

• airbyte

  51 10,796 10.0 Python

  Data integration platform for ELT pipelines from APIs, databases & files to warehouses & lakes.

  

  Project mention: airbyte VS cloudquery - a user suggested alternative | libhunt.com/r/airbyte | 2023-06-02

• great_expectations

  3 8,410 9.9 Python

  Always know what to expect from your data.

  

  Project mention: Data Quality at Scale with Great Expectations, Spark, and Airflow on EMR | dev.to | 2023-04-24

  Great Expectations (GE) is an open-source data validation tool that helps ensure data quality.

• Kedro

  10 8,408 8.1 Python

  A Python framework for creating reproducible, maintainable and modular data science code.

  

  Project mention: A Polars exploration into Kedro | dev.to | 2023-05-17

  # pyproject.toml [project] dependencies = [ "kedro @ git+https://github.com/kedro-org/[email protected]", "kedro-datasets[pandas.CSVDataSet,polars.CSVDataSet] @ git+https://github.com/kedro-org/[email protected]#subdirectory=kedro-datasets", ]

• papermill

  7 5,251 6.6 Python

  📚 Parameterize, execute, and analyze notebooks

  

  Project mention: Show HN: Mercury – convert Jupyter Notebooks to Web Apps without code rewriting | news.ycombinator.com | 2023-06-02

  I'm using Papermill to operationalize Notebooks (https://github.com/nteract/papermill), it e.g. also has airflow support. I'm really happy with papermill for automatic notebook execution, in my field it's nice that we can go very quickly from analysis to operations -- while having super transparent "logging" in the executed notebooks.

• pipelines

  0 3,205 9.7 Python

  Machine Learning Pipelines for Kubeflow

  

• InfluxDB

  www.influxdata.com

  sponsored

  Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  

• towhee

  5 2,250 9.7 Python

  Towhee is a framework that is dedicated to making neural data processing pipelines simple and fast.

  

  Project mention: Welcome to generate your embeddings with Towhee | news.ycombinator.com | 2023-04-20

• PyFunctional

  0 2,180 2.2 Python

  Python library for creating data pipelines with chain functional programming

• mara-pipelines

  0 2,005 3.6 Python

  A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

  

• MLBox

  0 1,425 0.0 Python

  MLBox is a powerful Automated Machine Learning python library.

• pytorch-toolbelt

  1 1,395 6.4 Python

  PyTorch extensions for fast R&D prototyping and Kaggle farming

  Project mention: Surrender cards are being distributed to Russianforces: 'Your ticket to a peaceful life. Show this card to a Ukrainian soldier - it will save your life and help you get back home' On the back: a telegram chat & phone number they can contact 'to receive detailed support'. | /r/ukraine | 2022-09-12

  For what it's worth, the QR code points to a URI shortener domain "qrfy.mobi" which redirects to a cloudflare hosted page that contains telegram information relating to "nikolay_bodenko" and "chaos_admin" ("For advertising and cooperation"? Whatever that is supposed to mean), which led me to "rf200_now" and "rf200_nooow" (looks to get spammed around a bit), which is purportedly supportive of Ukraine.

• galaxy

  2 1,109 10.0 Python

  Data intensive science for everyone.

  

  Project mention: BIOINFORMATICS PROJECT | /r/bioinformatics | 2022-10-16

• toil

  0 839 8.9 Python

  A scalable, efficient, cross-platform (Linux/macOS) and easy-to-use workflow engine in pure Python.

  

• pypyr automation task runner

  0 523 3.3 Python

  pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

  

• whispers

  0 436 0.0 Python

  Identify hardcoded secrets in static structured text (by Skyscanner)

  

• bodywork

  0 421 6.7 Python

  ML pipeline orchestration and model deployments on Kubernetes, made really easy.

  

• versatile-data-kit

  25 338 6.4 Python

  Build, run and manage your data pipelines with Python or SQL on any cloud

  

  Project mention: Looking for a data blogger | /r/opensource | 2023-05-19

  Here's the project: https://github.com/vmware/versatile-data-kit

• karton

  0 320 1.3 Python

  Distributed malware processing framework based on Python, Redis and S3.

  

• fluids

  1 275 9.2 Python

  Fluid dynamics component of Chemical Engineering Design Library (ChEDL)

  Project mention: AbaCalc - Calculator for engineers, students & technicians | /r/react | 2022-08-02

  I’m doing similar things using an api with some Python tools in the background to do the heavy lifting with units and thermo/chemical calcs using the Caleb Bell’s libraries for thermo and fluids

• aws-lambda-handler-cookbook

  7 259 10.0 Python

  This repository provides a working, deployable, open source based, AWS Lambda handler and CDK Python code. This handler embodies Serverless best practices and has all the bells and whistles for a proper production ready handler.

  Project mention: AWS Lambda Cookbook — Elevate your handler’s code — Part 4 — Environment Variables | dev.to | 2023-04-03

  This AWS CDK code defines the variables of the schema ‘MyHandlerEnvVars’ and sets their values. Look specifically at ‘__add_get_lambda_integration’ function.

• forte

  0 220 7.3 Python

  Forte is a flexible and powerful ML workflow builder. This is part of the CASL project: http://casl-project.ai/

  

• pipeline-live

  1 199 10.0 Python

  Pipeline Extension for Live Trading

  

  Project mention: How would I start to recode an old python library only compatible with an older version of python (3.6) to become compatible with a newer python version (3.8+)? | /r/learnpython | 2022-11-06

• SaaSHub

  www.saashub.com

  sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Python Pipeline related posts

• Postgres to Postgres incremental (timestamp) sync etl tool
  1 project | /r/dataengineer | 30 May 2023
• How useful is Airbytes in production pipelines?
  1 project | /r/dataengineering | 29 May 2023
• Fondant: Easily build and share datasets for foundation model fine-tuning
  1 project | news.ycombinator.com | 23 May 2023
• Do we need Spark if we're just loading data from source to target? Trying to move away from Informatica
  1 project | /r/dataengineering | 16 May 2023
• Test
  1 project | /r/u_solidigm | 15 May 2023
• test
  1 project | /r/u_solidigm | 15 May 2023
• Recommendation for Pipeline TD Projects for Junior Role.
  3 projects | /r/vfx | 3 May 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 7 Jun 2023

  Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com

Do not miss the trending Python projects with our weekly report!