hudi VS dbt-expectations

Apache Iceberg is one of the three types of lakehouse, the other two are Apache Hudi and Delta Lake.

Structured, Semi-structured and Unstructured can be stored in one single format, a lakehouse storage format like Delta, Iceberg or Hudi (assuming those don't require low-latency SLAs like subsecond).

Apache Hudi

Try Apache Hudi, it is fully integrated with AWS and offers almost everything that you requested.

the corresponding kafka topics have 30d retention and I intend on having s3 sink connector for long term storage (open to other ideas here too, I noticed theres a hudi connector also)

Apache Hudi

You could have a look at Apache Hudi - especially if you're running your Data Pipelines using Spark or Flink.

Have not used Soda, but dbt indeed is pretty good especially when adding dbt-expectations

dbt-expectations

I highly.. highly.. recommend the dbt-expectations extension from Catologica for dbt. It's a port of Great Expectations, except you can quickly thunk it in your schema.yml's and have it run as part of your dbt test process. Super powerful and it's prevented us from shipping bad data many times.

I'm used to utilising dbt and defining my tests there (along with dbt-utils or https://github.com/calogica/dbt-expectations): I simply add a list item to a column definition and can already define a great number of tests without having to copy code. I can even extend the pre-defined using generic tests. Writing custom tests also integrates nicely. Additionally it's very convenient to tag tests or define a severity. The learning curve for a business engineer is almost flat as long as they know some SQL.

Use dbt's testing functionality during your transformations with catalogica/dbt-expectations (Great Expectations framework ported to dbt)

Check out dbt-expectations https://github.com/calogica/dbt-expectations

There are certain things I would like to add such as data quality, I can use something like dbt great expectations, but I am not sure how much more I should force it before getting an airflow setup..

[dbt Labs employee here]

Check out dbt-expectations package[1]. It's a port of the Great Expectations checks to dbt as tests. The advantage of this is you don't need another tool for these pretty standard tests, and can be early incorporated into dbt workflows.

[1] https://github.com/calogica/dbt-expectations

Also check out dbt-expectations that is a port of Great Expectations that greatly expands the configurable (non-assert) tests.