Serverless ETL using AWS Lambda, Pandas, PostgreSQL, AWS CDK

This page summarizes the projects mentioned and recommended in the original post on dev.to

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • covid-19-data

    A repository of data on coronavirus cases and deaths in the U.S.

  • We pull data from This link and this John Hopkins dataset

  • Klayers

    Python Packages as AWS Lambda Layers

  • Before anything, Huge shoutout to Keith Rozario for providing us with this awesome Repo. where he defined layers for lambda functions to add libraries that doesn't exist by default in the lambda runtime without you having to create them yourself. I used his repo to add the requests, pandas and psycopg2 libraries to the project.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • configure-aws-credentials

    Configure AWS credential environment variables for use in other GitHub Actions.

  • The only commands we need to launch our infrastructure are cdk diff to detect changes and cdk deploy to commit changes. we can wrap these commands in a nice and easy Github actions workflow as after having configured your AWS access keys in Github. here's a link explaining how its done.

  • serverless-etl-cdk

    Deploy python code for ETL using the AWS CDK

  • Modern cloud infrastructure is being pushed to entirely new limits and possibilities and its only up to your imagination how you will use it. I have enjoyed a lot and learned so much while creating this. all the code for this article can be found in this Repo

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts