How To Start Your Next Data Engineering Project

• Apache Spark

  40 33,221 10.0 Scala

  Apache Spark - A unified analytics engine for large-scale data processing

  

  Apache Spark

• Druid

  8 11,830 9.7 Java

  Apache Druid: a high performance real-time analytics database.

  

  Apache Druid

• SonarLint

  www.sonarlint.org

  sponsored

  Deliver Cleaner and Safer Code - Right in Your IDE of Choice!. SonarLint is a free and open source IDE extension that identifies and catches bugs and vulnerabilities as you code, directly in the IDE. Install from your favorite IDE marketplace today.

  

• dagster

  15 4,908 9.9 Python

  An orchestration platform for the development, production, and observation of data assets.

  

  These are not the only choices, by any stretch of the imagination. Other popular options for orchestration are Dagster, *and Prefect. We actually recommend starting with *Airflow and then looking to others as you get more familiar with the processes.

• d3

  110 101,579 6.7 JavaScript

  Bring data to life with SVG, Canvas and HTML. :bar_chart::chart_with_upwards_trend::tada:

  

  D3.js

• nodejs-bigquery

  17 372 7.9 TypeScript

  Node.js client for Google Cloud BigQuery: A fast, economical and fully-managed enterprise data warehouse for large-scale data analytics.

  

  If you wanted to upgrade that idea, track down articles relating to that swing for discussion and post those. There is definite value in that data, and it is a pretty simple thing to do. You are just using a Cloud Composer to ingest the data and storing it in a data warehouse like BigQuery or Snowflake, creating a Twitter bot to post outputs to Twitter using something like Airflow.

• Airflow

  67 26,419 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

  Airflow

Suggest a related project