data-integration

Open-source projects categorized as data-integration
Edit details
Language: + Python + Go + Java + Rust + JavaScript + TypeScript + Jupyter Notebook
Topics: ETL data-engineering Data Python elt
Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
Sponsored

Top 23 data-integration Open-Source Projects

  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    • Project mention: Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions | dev.to | 2024-02-12

    Contributing to Apache Airflow's open-source project immersed me in collaborative coding. Experienced maintainers rigorously reviewed my contributions, providing constructive feedback. This ongoing dialogue refined the codebase and honed my understanding of best practices.

  • airbyte

    The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.

    • Project mention: Launch HN: Bracket (YC W22) – Two-Way Sync Between Salesforce and Postgres | news.ycombinator.com | 2023-12-12

    I'l also give a shout-out to Airbyte (https://airbyte.com/), with which I've had some limited success with integrating Salesforce to a local database. The particular pull for Airbyte is that we can self-host the open source version, rather than pay Fivetran a significant sum to do this for us.

    It's an immature tool, so I don't yet know that I can claim we've spent _less_ than Fivetran on the additional engineering and ops time, but it feels like it has potential to do so once stabilized.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • dagster

    An orchestration platform for the development, production, and observation of data assets.

    • Project mention: Experience with Dagster.io? | news.ycombinator.com | 2023-07-25

  • seatunnel

    SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.

    • Project mention: SeaTunnel – super high-performance, distributed data integration tool | news.ycombinator.com | 2024-04-28

  • Mage

    🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

    • Project mention: FLaNK AI-April 22, 2024 | dev.to | 2024-04-22

  • kestra

    Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

    • Project mention: A High-Performance, Java-Based Orchestration Platform | /r/java | 2023-10-11

    Kestra's communication is asynchronous and based on a queuing mechanism. It leverages the Micronaut framework and offers two runners: one that uses a database (JDBC) for both the message queue and resource storage, and another that uses Kafka as the message queue and Elasticsearch as the resource storage. The platform is fully extensible and plugin-based, providing a rich set of plugins for various workflow tasks, triggers, and data storage options. For those interested, the GitHub repository is available here: https://github.com/kestra-io/kestra

  • cloudquery

    The open source high performance ELT framework powered by Apache Arrow

    • Project mention: We might want to regularly keep track of how important each server is | news.ycombinator.com | 2024-02-06

    Check out CloudQuery - https://github.com/cloudquery/cloudquery for an easy cloud asset inventory.

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo

  • hudi

    Upserts, Deletes And Incremental Processing on Big Data.

    • Project mention: Getting Started with Flink SQL, Apache Iceberg and DynamoDB Catalog | dev.to | 2023-12-18

    Apache Iceberg is one of the three types of lakehouse, the other two are Apache Hudi and Delta Lake.

  • Rudderstack

    Privacy and Security focused Segment-alternative, in Golang and React

    • Project mention: Rudderstack Switches to Elastic License | news.ycombinator.com | 2023-09-08

  • jitsu

    Jitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days

  • paradedb

    Postgres for Search and Analytics

    • Project mention: Using ClickHouse to scale an events engine | news.ycombinator.com | 2024-04-11

  • awesome-single-cell

    Community-curated list of software packages and data resources for single-cell, including RNA-seq, ATAC-seq, etc.

  • fluvio

    Lean and mean distributed stream processing system written in rust and web assembly.

    • Project mention: Ask HN: WebSocket Relay? | news.ycombinator.com | 2024-02-27

  • incubator-devlake

    Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

  • ingestr

    ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

    • Project mention: FLaNK 04 March 2024 | dev.to | 2024-03-04

  • mara-pipelines

    A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow

  • bitsail

    BitSail is a distributed high-performance data integration engine which supports batch, streaming and incremental scenarios. BitSail is widely used to synchronize hundreds of trillions of data every day.

  • kuwala

    Kuwala is the no-code data platform for BI analysts and engineers enabling you to build powerful analytics workflows. We are set out to bring state-of-the-art data engineering tools you love, such as Airbyte, dbt, or Great Expectations together in one intuitive interface built with React Flow. In addition we provide third-party data into data science models and products with a focus on geospatial data. Currently, the following data connectors are available worldwide: a) High-resolution demograp

    • Project mention: Show HN: GeoSage – A ETL Webtool for Geo and Demographics Data from the Open Web | news.ycombinator.com | 2023-10-05

    --> Google Trends Data for Regions (Coming Soon)

    The tool goes beyond our previously published CLI tool (https://github.com/kuwala-io/kuwala/tree/master/kuwala) by providing a hostable solution with a user-friendly interface. We have not open-sourced it yet but a demo is available here: https://geosage.kuwala.io/.

    Urban planners can utilize movement data to analyze foot traffic in different city zones. Marketers can leverage demographic data to tailor campaigns more effectively. Developers can build their apps on top of it.

    To round it up .... GeoSage brings...

    Unified Data Management: Access data from OSM, Facebook, and soon Google, all in one place.

  • transfer

    Database replication platform that leverages change data capture. Stream production data from databases to your data warehouse (Snowflake, BigQuery, Redshift) in real-time.

    • Project mention: Migrate mongodb Datawarehouse to snowflake | /r/snowflake | 2023-12-04

  • conduit

    Conduit streams data between data stores. Kafka Connect replacement. No JVM required. (by ConduitIO)

    • Project mention: Pulling CDC data from Postgres | /r/dataengineering | 2023-04-30

    I'd like to mention Conduit + its Postgres connector. The Pg connector comes built-in, so all that is needed is a single Conduit binary to get started. It relies on WAL, but the connector creates the replication slot itself (if needed).

  • scarches

    Reference mapping for single-cell genomics

  • recap

    Work with your web service, database, and streaming schemas in a single format.

    • Project mention: Recap: A python library for describing database tables and serialization formats with minimal type coercion. | /r/dataengineering | 2023-07-12

    The Github Repo: https://github.com/recap-build/recap

  • cuelake

    Use SQL to build ELT pipelines on a data lakehouse.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

data-integration related posts

Index

What are some of the best open-source data-integration projects? This list will help you:

Project Stars
1 Airflow 34,485
2 airbyte 14,054
3 dagster 10,215
4 seatunnel 7,223
5 Mage 7,001
6 kestra 6,340
7 cloudquery 5,584
8 hudi 5,066
9 Rudderstack 3,926
10 jitsu 3,845
11 paradedb 3,863
12 awesome-single-cell 2,907
13 fluvio 2,638
14 incubator-devlake 2,430
15 ingestr 2,308
16 mara-pipelines 2,054
17 bitsail 1,576
18 kuwala 755
19 transfer 530
20 conduit 345
21 scarches 310
22 recap 307
23 cuelake 284

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com