Looking for open source projects that use data pipelines and big data flows

This page summarizes the projects mentioned and recommended in the original post on /r/dataengineering

Scout Monitoring - Free Django app performance insights with Scout Monitoring
Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.
www.scoutapm.com
featured
InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
  • meltano

    I know really sure if this is what are you looking for, but take a look at Meltano

  • Scout Monitoring

    Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

    Scout Monitoring logo
  • pipelinewise

    Data Pipeline Framework using the singer.io spec

  • nifi

    Apache NiFi

    apache nifi is a good match for what you are trying to find I think. It's also opensource so you can contribute as well.

  • Logstash

    Logstash - transport and process your logs, events, or other data

    Is logstash the kind of project you are looking for? https://github.com/elastic/logstash

  • jet-train

  • electricitymaps-contrib

    A real-time visualisation of the CO2 emissions of electricity consumption

  • grouparoo

    Discontinued 🦘 The Grouparoo Monorepo - open source customer data sync framework

    The product I'm working on called Grouparoo might be a good fit: https://github.com/grouparoo/grouparoo

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • FLaNK 25 December 2023

    33 projects | dev.to | 26 Dec 2023
  • 10+ Open-Source Projects For Web Developers In 2023

    14 projects | dev.to | 10 Apr 2023
  • Build a data ingestion pipeline using Kafka, Flink, and CrateDB

    6 projects | dev.to | 10 May 2021
  • Ask HN: Is there a way to hack YouTube landing page to see only certain topics?

    1 project | news.ycombinator.com | 23 Jul 2024
  • Multipass + Cloud-Init

    1 project | dev.to | 22 Jul 2024