Top 7 data-engineering-pipeline Open-Source Projects

Udacity-Data-Engineering-Projects

5 1,295 0.0 Python

Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

Project mention: Pitanje za data engineering? | /r/programiranje | 2023-06-30

versatile-data-kit

52 411 9.7 Python

One framework to develop, deploy and operate data workflows with Python and SQL.

Project mention: Looking for a data blogger | /r/opensource | 2023-05-19

Here's the project: https://github.com/vmware/versatile-data-kit

InfluxDB

www.influxdata.com featured

Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
prefect-deployment-patterns

1 93 0.0 Python

Code examples showing flow deployment to various types of infrastructure
Apache-Spark-Guide

2 28 1.8 Python

Apache Spark Guide
f1-data-pipeline

1 23 6.8 Python

F1 Data Pipeline

Project mention: Created my first Data Engineering Project which integrates F1 data using Prefect, Terraform, dbt, BigQuery and Looker Studio | /r/dataengineering | 2023-07-01

Github

business_closures_de_pipeline

1 14 0.0 Python

Data Engineering pipeline hosted entirely in the AWS ecosystem utilizing DocumentDB as the database
Shift

1 9 8.1 Go

Shift is a high performance better alternative to Airbyte, Singer, Meltano (by piyushsingariya)

Project mention: Alternative to Airbyte, Singer and Meltano | /r/dataengineering | 2023-08-11

As side hobby I started working on this personal project https://github.com/piyushsingariya/Kaku

SaaSHub

www.saashub.com featured

SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

data-engineering-pipeline related posts

✨ 5 Free Resources to Learn Data Engineering 🚀

3 projects | /r/dataengineering | 27 Sep 2022
Almost done with my DE Project. Mind Taking a look?

1 project | /r/dataengineering | 21 Dec 2020

Index

What are some of the best open-source data-engineering-pipeline projects? This list will help you:

	Project	Stars
1	Udacity-Data-Engineering-Projects	1,295
2	versatile-data-kit	411
3	prefect-deployment-patterns	93
4	Apache-Spark-Guide	28
5	f1-data-pipeline	23
6	business_closures_de_pipeline	14
7	Shift	9