Python data-pipeline

Open-source Python projects categorized as data-pipeline
Edit details
Topics: Python data-engineering Data Science Bigquery Machine Learning
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
Sponsored

Top 15 Python data-pipeline Projects

  • airbyte

    The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

    • Project mention: Launch HN: Bracket (YC W22) – Two-Way Sync Between Salesforce and Postgres | news.ycombinator.com | 2023-12-12

    I'l also give a shout-out to Airbyte (https://airbyte.com/), with which I've had some limited success with integrating Salesforce to a local database. The particular pull for Airbyte is that we can self-host the open source version, rather than pay Fivetran a significant sum to do this for us.

    It's an immature tool, so I don't yet know that I can claim we've spent _less_ than Fivetran on the additional engineering and ops time, but it feels like it has potential to do so once stabilized.

  • ingestr

    ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

    • Project mention: FLaNK 04 March 2024 | dev.to | 2024-03-04

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • doit

    task management & automation tool

    • Project mention: How do you deal with CI, project config, etc. falling out of sync across repos? | /r/ExperiencedDevs | 2023-12-06

    I like mage for Go and doit for Python.

  • DataEngineeringProject

    Example end to end data engineering project.

  • covalent

    Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments. (by AgnostiqHQ)

    • Project mention: Remote execution of code | /r/Python | 2023-12-05

    Pretty interesting request, if SSH is not used, i would try using something like dask which uses tcp to connect and execute assuming your workers are in another machine.I also think something like covalent can be used to extend your own custom plugin in their ecosystem to connect how you want. We have a very custom private plugin written on top of covalent's to have a custom protocol to connect our central on-prem GPU machines to our local laptops that is rpc based, mostly for high performance as well as some mandate security from where the GPU machines are. Once done it is pretty much something like

  • piperider

    Code review for data in dbt

    • Project mention: Show HN: PipeRider – open-source Data Impact Analysis for dbt changes | news.ycombinator.com | 2023-09-06

  • premier-league

    A Data Engineering project. Repository for backend infrastructure and Streamlit app files for a Premier League Dashboard.

    • Project mention: Google Cloud Portfolio Projects? | /r/googlecloud | 2023-12-09

    I have a data engineering project that uses BigQuery, Cloud Run, Compute Engine, Cloud SQL, Artifact Registry, Firestore, and Datastream.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • datajob

    Build and deploy a serverless data pipeline on AWS with no effort.

  • patterns-devkit

    Data pipelines from re-usable components

  • airflow-testing-ci-workflow

    (project & tutorial) dag pipeline tests + ci/cd setup

  • VQASynth

    Compose multimodal datasets 🎹

    • Project mention: Show HN: VQASynth – pipelines to synthesize VQA datasets | news.ycombinator.com | 2024-02-23

  • alto

    Alto is a versatile data integration tool that allows you to easily run Singer plugins, build and cache PEX files encapsulating those plugins, and create a data reservoir whereby you can extract once and replay to as many destinations as you want. (by z3z1ma)

  • datatap-python

    Focus on Algorithm Design, Not on Data Wrangling

  • data-engineer-challenge

    Challenge Data Engineer

  • pyDag

    Scheduling Big Data Workloads and Data Pipelines in the Cloud with pyDag

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

Python data-pipeline related posts

Index

What are some of the best open-source data-pipeline projects in Python? This list will help you:

Project Stars
1 airbyte 13,923
2 ingestr 2,308
3 doit 1,780
4 DataEngineeringProject 985
5 covalent 687
6 piperider 466
7 premier-league 142
8 datajob 107
9 patterns-devkit 106
10 airflow-testing-ci-workflow 81
11 VQASynth 71
12 alto 48
13 datatap-python 34
14 data-engineer-challenge 25
15 pyDag 24

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com