An orchestration platform for the development, production, and observation of data assets.
This is probably a bit early in your journey but I recommend Dagster . Its much like airflow but better in my opinion and doesnt require Docker setups to install and run which is very handy. But like I said its probably a bit early if you're not yet comfortable with Python. In the interim regarding automation i'd say go to basics and build from there, so say you've got a working script that downloads data and updates a table, just set up a Task scheduler job if you're on windows , cron if otherwise. Thats already step 1 in automation.
Truly a developer’s best friend. Scout APM is great for developers who want to find and fix performance issues in their applications. With Scout, we'll take care of the bugs so you can focus on building great things 🚀.
ETL advice appreciated
1 project | reddit.com/r/ETL | 19 Jun 2022
Thinking of making a switch from actuarial science to data engineering
1 project | reddit.com/r/dataengineering | 15 Apr 2022
Developing in Dagster
2 projects | dev.to | 25 Mar 2022
What is this kind of texture called?
1 project | reddit.com/r/Design | 19 Dec 2021
Airflow 2.0 vs Prefect
1 project | reddit.com/r/dataengineering | 20 Oct 2021