data_engineering_project_template VS public-api-lists

SCPs your code into EC2

Docker is used to containerize your application. For e.g. this Dockerfile is used to create a container and it specifies what OS it is, etc. You can run docker on any machine and you can think of it as running a separate os (not exactly, but close enough) on the machine. What Docker provides is the ability to replicate OS & its packages (e.g. python modules) across machines so that you don't run into "hey that worked on my computer" type issues.

I have a few e2e projects, if that might help. While they can be deployed to AWS via the included terraform file (costs money) you can run them locally with docker. I list the projects from simplest to more complicated 1. I’d recommend starting at https://www.startdataengineering.com/post/data-engineering-project-to-impress-hiring-managers/ this is the simplest. 2. Once you have it running, and get an overview of the components( docker, ec2, Postgres), then I’d recommend looking at this article https://www.startdataengineering.com/post/data-engineering-projects-with-free-template/ to understand how the components work together. 3. Try out the pipeline with a data source if your choosing. I use https://github.com/public-api-lists/public-api-lists to get some data API. 4. Once you get a good understanding of how data is pulled and loaded along with how it’s scheduled, then I’d recommend looking at this airflow project https://www.startdataengineering.com/post/data-engineering-project-for-beginners-batch-edition/ I posted about this a while back https://www.reddit.com/r/dataengineering/comments/ygieh8/data\_engineering\_projects\_with\_template\_airflow/ Hope this helps. LMK if you have any questions.

And then try out the pipeline with a data source if your choosing. I use https://github.com/public-api-lists/public-api-lists to get some data API.

Suggest to take a look through https://github.com/public-api-lists/public-api-lists

Sounds like you're stuck in tutorial purgatory. It's time to drop the training wheels and apply the knowledge you've learned to some personal projects. Pickup pygame, or build a web scraper. Here's a list of free public api's. This might get you started. https://github.com/public-api-lists/public-api-lists

Public-api-lists (fork of original one I linked)

public-api-lists Another great repository to find good APIs for development. It has more than 4K stars on GitHub. These APIs are more inclined towards the development side. It is also organized into suitable categories.