Evolutionary Data Infrastructure

This page summarizes the projects mentioned and recommended in the original post on dev.to
Database Workflow engine NodeJS change-data-capture Airflow

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured

  • kafka-connect-bigquery

    A Kafka Connect BigQuery sink connector

    • But is there a way to solve the schema problem? The answer is, yes, through KCBQ.

  • flink-statefun

    Apache Flink Stateful Functions

    • Therefore, I still recommend using a streaming framework such as Apache Flink or Apache Kafka Streams.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo

  • debezium

    Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.

    • We use Debezium to capture changes to each database and send the streams to Kafka, and later KCBQ subscribes to the Kafka streams and archives them to BigQuery.

  • nodejs-bigquery

    Node.js client for Google Cloud BigQuery: A fast, economical and fully-managed enterprise data warehouse for large-scale data analytics.

    • In addition, batch tasks require knowledge of the data schema of each service in order to get the data correctly and save it to the corresponding warehouse table. Assuming our data warehouse is GCP BigQuery, the schema in the warehouse table also needs to be created and modified manually.

  • Airflow

    Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

    • For batch processing, I recommend using Apache Airflow, which is easy to manage and easy to script for various DAGs to address the needs of multiple batch processing scenarios.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • Migrating to Snowflake, Redshift, or BigQuery? Use Datafold to Avoid these Common Pitfalls

    2 projects | dev.to | 15 Dec 2021

  • Data Analytics at Potloc I: Making data integrity your priority with Elementary & Meltano

    4 projects | dev.to | 5 Jan 2023

  • You can't leak users' data if you don't hold it

    2 projects | news.ycombinator.com | 30 Mar 2024

  • NodeJS Security Best Practices

    11 projects | dev.to | 19 Feb 2024

  • Building in Public: Leveraging Tublian's AI Copilot for My Open Source Contributions

    1 project | dev.to | 12 Feb 2024