Cookbook VS data-engineer-roadmap

https://github.com/andkret/Cookbook arunca un ochi aici. Omul are si youtube channel https://www.youtube.com/@andreaskayy

I can recommend https://learndataengineering.com by Anreas Krenz. Will guide you via all important topics starting from sql & python to building pipelines using AWS/GCP. I used to participate for 1 year (costs ~ 200 Euro/220$). It's a self-paced. So for ~15h/week you can switch into DE position for appr. 6 months.

I wonder if anyone involved in this post and comments have tried this? https://learndataengineering.com/

I think it's allowed: https://learndataengineering.com

From your post, you seem to be trained for data science for physics modeling, so I'd recommend to get started with https://ml-ops.org/ and for the data engineering part, I found this https://github.com/andkret/Cookbook open source cookbook to be invaluable.

I am doing this currently to fill in the blanks: https://learndataengineering.com. Also, do you know Python? If not take class on Udemy on that. Finally, data engineering is all about tools these days. I saw someone recommended this book here: Data Engineering with Python, I find it super hopeful. You download these tools (Apache Airflow, etc) and get a go with it. I am going to build some data pipelines via this book :)

the best way to be a dev or DE is to build stuff, not learning about algorithms. Just google DE academy, bootcamp or so. The linked one is quite good for a cheap price. A degree prepares you mostly for a PhD, not for a job. So dont look for degrees preparing you for a job in general.

I usually don't do self promotion, but because you directly asked for a good source. Look at my academy: https://learndataengineering.com

Find something like https://learndataengineering.com/, udemy or any other 'bootcamp/course' that goes on for few months and learn it. It is important that you will have some mentors or study buddies to exchange ideas or so.

In DevOps tools I've worked with GitHub + Jenkins, GitLab + k8s, and I'm now primarily working in the Argo Stack. Depending on where you're at technically, you might use something different. IaC is a ust as well, maybe some config management. Generally I've found that as a Data Engineer with a lot of infra/CICD knowledge, I generally get pigeonholed into those positions on a team, so be prepared for that. I really like this roadmap for DevOps , so you can see where your tech skills are at currently, and what you may need to learn. On top of that, you'll need to learn some data tools. Airflow + dbt is hot right now, Argo is sometimes used in MLOps, Azure Data Stack (I'm not familiar with it) seems common, and probably Spark in almost all cases. You can also checkout in visualization tools probably further down the line, I generally stick to something free when learning on my own, Superset or Google Data Studio (Might be Looker Studio now? Not sure, it's been a while). Here's a roadmap for DE too. I love these roadmaps for getting started, but don't let them distract you from exploring a path more appropriate to what you want to achieve. Generally I've found that as a Data Enigneer with a lot of infra/CICD knowledge, I generally get pigeonholed into those positions on a team

Anyways, sorry bit of a rant - I land somewhere in the middle. I would say take formal classes and resources when you can. If you have access to a free course a semester, that's incredible in my opinion. If I were in your shoes, I would follow a roadmap and see if there are courses that check off a box in that roadmap. So for example, you know you need to learn CS fundamentals - see if you can take a DSA class or something. Or take a class on databases. Or an OOP or databases class. I would take those classes if I had the opportunity just because I didn't when I was in college. No one course will check every box for sure.

I don't remember all of the name of the courses but I think this roadmap can put you in the right direction https://github.com/datastacktv/data-engineer-roadmap

You can follow this roadmap https://github.com/datastacktv/data-engineer-roadmap I have already replied some redditors with suggestions, you can read them.