Built and automated a complete end-to-end ELT pipeline using AWS, Airflow, dbt, Terraform, Metabase and more as a beginner project!

• terraform

  75 41,053 9.9 Go

  Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

  

  Thanks! I configured my AWS account credentials through the AWS CLI first. Then I used Terraform to create all the services on AWS. You can find all the files relating to setting up the cloud environment under `terraform/`. You can create GCP resources through Terraform as well. All the resources used in the project come under the AWS free tier, so I didn't incur any costs throughout the entire project!

• audiophile-e2e-pipeline

  0 170 0.0 Python

  Pipeline that extracts data from Crinacle's Headphone and InEarMonitor databases and finalizes data for a Metabase Dashboard.

• InfluxDB

  www.influxdata.com

  sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• pydantic

  12 18,442 9.8 Python

  Data validation using Python type hints

  

  Initial data parsing and validation through Pydantic to generate silver data.

• Airflow

  16 34,317 10.0 Python

  Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

  

  Infrastructure provisioning through Terraform, containerized through Docker and orchestrated through Airflow. Created dashboard through Metabase.

Suggest a related project