data-orchestration

Open-source projects categorized as data-orchestration
Edit details
Language: + Python + Java + C++
Topics: Spark data-engineering ETL Data Data Analysis
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

Top 5 data-orchestration Open-Source Projects

  • Mage

    🧙 The modern replacement for Airflow. Mage is an open-source data pipeline tool for transforming and integrating data. https://github.com/mage-ai/mage-ai

    • Project mention: FLaNK AI-April 22, 2024 | dev.to | 2024-04-22

  • Alluxio (formerly Tachyon)

    Alluxio, data orchestration for analytics and machine learning in the cloud

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • kestra

    Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.

    • Project mention: A High-Performance, Java-Based Orchestration Platform | /r/java | 2023-10-11

    Kestra's communication is asynchronous and based on a queuing mechanism. It leverages the Micronaut framework and offers two runners: one that uses a database (JDBC) for both the message queue and resource storage, and another that uses Kafka as the message queue and Elasticsearch as the resource storage. The platform is fully extensible and plugin-based, providing a rich set of plugins for various workflow tasks, triggers, and data storage options. For those interested, the GitHub repository is available here: https://github.com/kestra-io/kestra

  • incubator-graphar

    An open source, standard data file format for graph data storage and retrieval.

  • data-aware-orchestration

    Data-aware orchestration with dagster, dbt, and airbyte

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The open source projects on this list are ordered by number of github stars. The number of mentions indicates repo mentiontions in the last 12 Months or since we started tracking (Dec 2020).

data-orchestration related posts

  • GraphAr Release v0.5.0

    1 project | news.ycombinator.com | 16 May 2023

  • Show HN: GraphAr – An File Format for Graph Data Archiving/Exchanging

    1 project | news.ycombinator.com | 3 May 2023

  • Show HN: GraphAr – Open-source file format for archiving/exchanging graph data

    1 project | news.ycombinator.com | 27 Apr 2023

  • Show HN: GraphAr – Open-source file format for archiving/exchanging graph data

    1 project | news.ycombinator.com | 18 Apr 2023

  • Mage Battlegrounds: Craft insights from real-time customer behavior analysis

    2 projects | dev.to | 10 Apr 2023

  • Looking for an open-source project

    2 projects | /r/dataengineering | 13 Feb 2023

  • Daskqueue: Dask-based distributed task queue

    1 project | /r/dataengineering | 5 Feb 2023
  • A note from our sponsor - InfluxDB
    www.influxdata.com | 17 May 2024
    Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →

Index

What are some of the best open-source data-orchestration projects? This list will help you:

Project Stars
1 Mage 7,131
2 Alluxio (formerly Tachyon) 6,667
3 kestra 6,605
4 incubator-graphar 177
5 data-aware-orchestration 28

Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com