Data pipeline suggestions

• airbyte

  139 13,923 10.0 Python

  The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

  

Ingestion / Extraction: Airbyte, Singer, Jitsu

• jitsu

  13 3,831 9.8 TypeScript

  Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

  

Ingestion / Extraction: Airbyte, Singer, Jitsu

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• dbt-core

  86 8,881 9.7 Python

  dbt enables data analysts and engineers to transform their data using the same practices that software engineers use to build applications.

  

Transformation: dbt

• Airflow

  169 34,397 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

Orchestration: Airflow, Dagster

• dagster

  46 10,173 10.0 Python

  An orchestration platform for the development, production, and observation of data assets.

  

Orchestration: Airflow, Dagster

• great_expectations

  15 9,440 9.9 Python

  Always know what to expect from your data.

  

Testing: GreatExpectations

• monosi

  20 320 0.0 Python

  Open source data observability platform

  

Observability: Monosi

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• grouparoo

  27 607 9.9 JavaScript

  Discontinued 🦘 The Grouparoo Monorepo - open source customer data sync framework

  

Reverse ETL: Grouparoo, Castled

• castled

  12 316 9.9 Java

  Discontinued Castled is an open source reverse ETL solution that helps you to periodically sync the data in your db/warehouse into sales, marketing, support or custom apps without any help from engineering teams

  

Reverse ETL: Grouparoo, Castled

• lightdash

  13 3,388 10.0 TypeScript

  Self-serve BI to 10x your data team ⚡️

  

Visualization / Analysis: Lightdash, Superset

• superset

  137 58,737 9.9 TypeScript

  Apache Superset is a Data Visualization and Data Exploration Platform

  

Visualization / Analysis: Lightdash, Superset

• ploomber

  121 3,369 7.8 Python

  The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

  

Check out Ploomber, (disclaimer: I'm the author) it has a simple API, and you can export to Airflow, AWS, Kubernetes. Supports all databases that work with Python and you can seamlessly transfer from a SQL step to a Python step. Here's an example.

• projects

  19 77 4.7 Jupyter Notebook

  Sample projects using Ploomber. (by ploomber)

  

Check out Ploomber, (disclaimer: I'm the author) it has a simple API, and you can export to Airflow, AWS, Kubernetes. Supports all databases that work with Python and you can seamlessly transfer from a SQL step to a Python step. Here's an example.

Suggest a related project