Data engineering projects with template: Airflow, dbt, Docker, Terraform (IAC), Github actions (CI/CD) & more

• data_engineering_project_template

  3 176 2.2 HCL

  A template repository to create a data project with IAC, CI/CD, Data migrations, & testing

Docker is used to containerize your application. For e.g. this Dockerfile is used to create a container and it specifies what OS it is, etc. You can run docker on any machine and you can think of it as running a separate os (not exactly, but close enough) on the machine. What Docker provides is the ability to replicate OS & its packages (e.g. python modules) across machines so that you don't run into "hey that worked on my computer" type issues.

• black

  322 37,376 9.4 Python

  The uncompromising Python code formatter

  

Formatting: isort & black

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• terraform

  500 41,118 9.9 Go

  Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

  

IAC: Terraform

• Flake8

  33 3,257 7.5 Python

  flake8 is a python tool that glues together pycodestyle, pyflakes, mccabe, and third-party plugins to check the style and quality of some python code.

  

Lint check: flake8

• isort

  41 6,314 7.4 Python

  A Python utility / library to sort imports.

  

Formatting: isort & black

• Docker Compose

  386 32,367 9.6 Go

  Define and run multi-container applications with Docker

  

local development: Docker & Docker compose

• public-api-lists

  17 8,780 8.0 Python

  A collective list of free APIs for use in software and web development 🚀

  

And then try out the pipeline with a data source if your choosing. I use https://github.com/public-api-lists/public-api-lists to get some data API.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project