The Data Engineering Docker-Compose Starter Kit

• blog_de_starter_kit

  1 2 5.1 Python

Here you can find my starter kit that includes the following services/tools:

• docs.getdbt.com

  2 106 10.0 JavaScript

  The code behind docs.getdbt.com

  

Consider adding a Python producer service or producing messages directly from Mage. With Mage, you can even set a trigger to launch the pipeline for every new file on s3. Interested in running a dbt project? You can do it in the Mage pipeline on top of the MySQL database. If MySQL isn’t your favorite, swap it out for Postgres or Mongo. Essentially, there’s a wide range of options for adjusting and enhancing these services.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• awesome-public-datasets

  78 58,470 5.1

  A topic-centric list of HQ open datasets.

  

The “data.csv” file contains historical weather data from Tel Aviv, sourced from another article I wrote. If you wish, you can swap it with a public dataset, for example, from here.

• Docker Compose

  388 32,417 9.6 Go

  Define and run multi-container applications with Docker

  

Hint: If you don’t know what Docker and Docker Compose are, now’s the perfect time to go find out.

Suggest a related project