hudi VS missing-semester

Apache Iceberg is one of the three types of lakehouse, the other two are Apache Hudi and Delta Lake.

Structured, Semi-structured and Unstructured can be stored in one single format, a lakehouse storage format like Delta, Iceberg or Hudi (assuming those don't require low-latency SLAs like subsecond).

Apache Hudi

Try Apache Hudi, it is fully integrated with AWS and offers almost everything that you requested.

the corresponding kafka topics have 30d retention and I intend on having s3 sink connector for long term storage (open to other ideas here too, I noticed theres a hudi connector also)

Apache Hudi

You could have a look at Apache Hudi - especially if you're running your Data Pipelines using Spark or Flink.

You mentioned having issues with accessory concepts so perhaps this might help: https://missing.csail.mit.edu/. There's also a chapter on git

https://missing.csail.mit.edu/ is a good way to pass the Dec-Jan break if you want to prep for CS2030S + some more general stuff.

Reflecting a little bit more I don't think it was replit's fault, per-say. But that change should have been made together with a larger adjustment to the program. Like adding a class/unit in the style of [the missing semester](https://missing.csail.mit.edu/) to make sure people came away with a good range of intuitions.

From MJD's post: I think CS curricula should have a class that focuses specifically on these issues, on the matter of how do you actually write software?

But they never do.

FWIW, MIT's "The Missing Semester of Your CS Education" attempts to deal with this lack, though, even there, it's an unofficial course taught between terms, during MIT's IAP -- Independent Activities Period[1] -- and not an actual CS course.

[0] https://missing.csail.mit.edu/

[1] https://en.wikipedia.org/wiki/Traditions_and_student_activit...

Link to the "missing semester of your CS degree" course by MIT.

Rightly called The Missing Semester (of Your CS Education), this class from MIT will teach you how to use some of the tools that are fundamental to the software engineering ecosystem. From shell scripting to the fundamentals of information security—spanning around 12 lectures—you can add a bunch of practical skills to your toolbox.

Also highly suggest going through the missing semester