Top 15 etl-framework Open-Source Projects

• Logstash

  5 14,014 9.6 Java

  Logstash - transport and process your logs, events, or other data

  

• cloudquery

  102 5,591 10.0 Go

  The open source high performance ELT framework powered by Apache Arrow

  

Project mention: We might want to regularly keep track of how important each server is | news.ycombinator.com | 2024-02-06

Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• hamilton

  21 1,373 9.8 Jupyter Notebook

  Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage and metadata. Runs and scales everywhere python does.

  

Project mention: Show HN: Hamilton's UI – observability, lineage, and catalog for data pipelines | news.ycombinator.com | 2024-05-02

• getting-started

  16 1,220 0.0 Makefile

  This repository is a getting started guide to Singer. (by singer-io)

  

Project mention: Why do companies still build data ingestion tooling instead of using a third-party tool like Airbyte? | /r/dataengineering | 2023-12-06

Coincidently, I saw a presentation today on a nice half-way-house solution: using embeddable Python libraries like Sling and dlt - both open-source. See https://www.youtube.com/watch?v=gAqOLgG2iYY There is also singer.io which is more of a protocol than a library, but can also be installed although it looks like it is a true community effort and not so well maintained.

• quokka

  23 1,084 8.3 Python

  Making data lake work for time series (by marsupialtail)

Project mention: How Query Engines Work | news.ycombinator.com | 2023-09-08

An awesome read!

Something related that I found out about from HN a few months back is another engine called quokka. It's particularly interesting and applicable how quokka schedules distributed queries to outperform Spark https://github.com/marsupialtail/quokka/blob/master/blog/why...

• Cinchoo ETL

  0 738 6.3 C#

  ETL framework for .NET (Parser / Writer for CSV, Flat, Xml, JSON, Key-Value, Parquet, Yaml, Avro formatted files)

• metorikku

  0 576 2.4 Scala

  A simplified, lightweight ETL Framework based on Apache Spark

  

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

• flow

  2 352 9.9 PHP

  Flow PHP - data processing framework (by flow-php)

  

Project mention: Flow PHP: the first and most advanced PHP ETL framework | news.ycombinator.com | 2024-04-16

• kgtk

  1 341 4.3 Jupyter Notebook

  Knowledge Graph Toolkit

  

• dataall

  1 210 9.4 Python

  A modern data marketplace that makes collaboration among diverse users (like business, analysts and engineers) easier, increasing efficiency and agility in data projects on AWS.

  

• patterns-devkit

  5 106 2.9 Python

  Data pipelines from re-usable components

  

• csvplus

  0 66 0.0 Go

  csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

• Shift

  1 9 8.1 Go

  Shift is a high performance better alternative to Airbyte, Singer, Meltano (by piyushsingariya)

Project mention: Alternative to Airbyte, Singer and Meltano | /r/dataengineering | 2023-08-11

As side hobby I started working on this personal project https://github.com/piyushsingariya/Kaku

• flowrunner

  3 8 5.9 Python

  Flowrunner is a lightweight package to organize and represent Data Engineering/Science workflows

• DataPowerTools

  2 8 6.7 C#

  Bridging the gap between IEnumerable and IDataReader for dealing with unstructured and loosely-structured data, plus fast ETL + SQL Bulk Copy.

Project mention: Recommended patterns or tools for data/row migration between databases? | /r/dotnet | 2023-06-22

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

etl-framework related posts

• Flow PHP: the first and most advanced PHP ETL framework

  1 project | news.ycombinator.com | 16 Apr 2024

• FLaNK Weekly 31 December 2023

  25 projects | dev.to | 31 Dec 2023

• Why do companies still build data ingestion tooling instead of using a third-party tool like Airbyte?

  1 project | /r/dataengineering | 6 Dec 2023

• SymmetricDS: Open-Source, cross platform database replication software

  3 projects | news.ycombinator.com | 6 Aug 2023

• Breakthrough in the book search field! Use Apache SeaTunnel to improve the efficiency of book title similarity search

  3 projects | dev.to | 3 Jul 2023

• Quokka – Distributed Polars on Ray

  1 project | news.ycombinator.com | 30 Jun 2023

• Questions Regarding design DW

  1 project | /r/dataengineering | 24 Jun 2023
• A note from our sponsor - InfluxDB
  www.influxdata.com | 9 May 2024

  Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com