etl-pipeline

Open-source projects categorized as etl-pipeline
Edit details
Language: + Python + TypeScript + Jupyter Notebook + Clojure + Scala + PHP + Go
Topics: ETL data-engineering Python etl-framework Data Science
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured

Top 21 etl-pipeline Open-Source Projects

  • orchest

    Build data pipelines, the easy way 🛠️

  • pyspark-example-project

    Implementing best practices for PySpark ETL jobs and applications.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • hamilton

    Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

    • Project mention: Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines | news.ycombinator.com | 2024-05-02

  • Udacity-Data-Engineering-Projects

    Few projects related to Data Engineering including Data Modeling, Infrastructure setup on cloud, Data Warehousing and Data Lake development.

    • Project mention: Pitanje za data engineering? | /r/programiranje | 2023-06-30

  • tech.ml.dataset

    A Clojure high performance data processing system

  • metorikku

    A simplified, lightweight ETL Framework based on Apache Spark

  • flow

    Flow PHP - data processing framework (by flow-php)

    • Project mention: Flow PHP: the first and most advanced PHP ETL framework | news.ycombinator.com | 2024-04-16

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo

  • pipebird

    Discontinued Pipebird is open source infrastructure for securely sharing data with customers.

  • data-engineering-project-template

    This is a template you can use for your next data engineering portfolio project.

  • patterns-devkit

    Data pipelines from re-usable components

  • unstract

    No-code LLM Platform to launch APIs and ETL Pipelines to structure unstructured documents

    • Project mention: Ask HN: Is RAG the Future of LLMs? | news.ycombinator.com | 2024-04-14

    Fast changing libraries are a huge pain. That's why a no-code approach like Unstract (https://github.com/zipstack/unstract) makes sense.

  • uber-expenses-tracking

    The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift and Power BI.

  • prism

    Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python. (by runprism)

    • Project mention: Prism: the easiest way to create robust data workflows. Accessible via CLI | /r/coolgithubprojects | 2023-09-21

  • csvplus

    csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

  • bitcoinMonitor

    Near real time ETL to populate a dashboard.

    • Project mention: Best place to learn APIS? | /r/dataengineering | 2023-06-26

    I have a sample code here that pulls data from an API and loads it into a DB, scheduled by cron, that can help with some ideas.

  • Spooq

  • dados-censup

    Automação da ingestão de dados disponibilizados pelo INEP referente ao censo superior da educacão brasileira.

  • pipeline-docs-data-extractor

    (Let's build a) Robust pipeline for extracting structured data from various documents

    • Project mention: ETL Texts | news.ycombinator.com | 2024-01-14

  • workshop-realtime-data-pipelines

    You will inspect and run a sample architecture making use of Apache Pulsar™ and Pulsar Functions for real-time, event-streaming-based data ingestion, cleaning and processing.

  • ticker_selection_BI_dashboard

    Data Engineering Project: 4 shares of a stock data extraction, upload on MySql used to be in a BI project

  • reddit_api_elt

    • Project mention: Reddit ELT Pipeline | /r/dataengineering | 2023-12-11

    Hi everyone, this is my first DE project. Baitur5/reddit_api_elt (github.com) . It is basically about a data pipeline that extracts Reddit data for a Google Data Studio report, focusing on a specific subreddit Can you guys check it out , and give some advice & tips on how to improve it or the next things I should add.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

etl-pipeline related posts

Index

What are some of the best open-source etl-pipeline projects? This list will help you:

Project Stars
1 orchest 4,021
2 pyspark-example-project 1,370
3 hamilton 1,379
4 Udacity-Data-Engineering-Projects 1,295
5 tech.ml.dataset 639
6 metorikku 576
7 flow 355
8 pipebird 167
9 data-engineering-project-template 112
10 patterns-devkit 106
11 unstract 141
12 uber-expenses-tracking 94
13 prism 79
14 csvplus 66
15 bitcoinMonitor 55
16 Spooq 8
17 dados-censup 6
18 pipeline-docs-data-extractor 5
19 workshop-realtime-data-pipelines 3
20 ticker_selection_BI_dashboard 2
21 reddit_api_elt 2

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com