analytics VS dbt-unit-testing

Take a look at gitlab's dbt project: https://gitlab.com/gitlab-data/analytics/-/blob/master/transform/snowflake-dbt/models/common/schema.yml

A good reference is the Gitlab data team repo. https://gitlab.com/gitlab-data/analytics

For example, their Data Team have a public repository, with a bunch of information on how they organize DAGs, machine learning projects, system configuration, etc.

So I've run through the DBT tutorial stuff and looked over some fairly complex uses of it i.e. GitLab Data and I was wondering if anyone has any opinions or insights into the use of jinja templating in the sql?

Gitlab has their DBT repo open source and is very useful for seeing how to structure a project at scale. https://gitlab.com/gitlab-data/analytics/-/tree/master/transform/snowflake-dbt

Currently the team is working hard on this: https://gitlab.com/gitlab-data/analytics/-/issues/9508

GitLab's dbt project is an excellent example of a mature project at scale. They also have a comprehensive guide to their methodology.

If you use dbt for transformations Dbt Unit Testing (https://github.com/EqualExperts/dbt-unit-testing) is getting some attention (https://www.thoughtworks.com/radar/languages-and-frameworks?blipid=202304042)

dbt-unit-testing

We use this and take an example-based tests approach for any non-trivial tables: https://github.com/EqualExperts/dbt-unit-testing

> How do you test some SQL logic in isolation?

I do this using sql

1. Extracting an 'ephemeral model' to different model file

2. Mock out this model in upstream model in unit tests https://github.com/EqualExperts/dbt-unit-testing

3. Write unit tests for this model.

This is not different than regular software development in a language like java.

I would argue its even better better because unit tests are always in tabular format and pretty easy to understand. Java unit tests on other hand are never read by devs in practice.

I haven't done it yet but there are some popular blogs as well as a DBT package someone created.

> I really don’t understand the communities obsession with unwieldy tools like DBT.

It lets me write test first sql transforms. I never thought TDD sql would be possible. My sql is so much more readable with common logic extracted into ephmeral models. I practice same method to write clear code to write sql, eg: too many mocks = refactor into separate model ( class) .

I think DBT made this possible with refs that can be swapped out with mocks. This is the awesome library I am using https://github.com/EqualExperts/dbt-unit-testing