nessie VS Mermaid

Nessie is an innovative open-source catalog that extends beyond the traditional catalog capabilities in the Apache Iceberg ecosystem, introducing git-like features to data management. This catalog not only tracks table metadata but also allows users to capture commits at a holistic level, enabling advanced operations such as multi-table transactions, rollbacks, branching, and tagging. These features provide a new layer of flexibility and control over data changes, resembling version control systems in software development.

Try Nessie https://github.com/projectnessie/nessie - it recently got trino support as well ..

Project Nessie (https://projectnessie.org/) will be the catalog that eventually decouples Iceberg from Hive. At that point, I think it will be a no brainer to go Iceberg over Delta.

The Dremio Sonar query engine can query your data where it exists whether it's AWS Glue, S3, Nessie Catalogs, MySQL, Postgres, RedShift and an ever growing list of sources.

We will also need a catalog to track all of these tables, with the open source Project Nessie we can do just that, and also get great versioning features similar to using Git when developing applications allowing data engineers to practice "data as code" and "write-audit-publish" patterns on their data.

Mermaid JS is a tool that lets you create diagrams and charts such as flowcharts, sequence diagrams, Gantt charts using simple text commands. It works by converting your text commands into a graphical representation that you can customize and share. https://mermaid.live/

I'm not OP, but I just ask GPT to turn code or process or whatever else into a mermaid diagram. Most of the time I don't even need to few-shot prompt it with examples. Then you dump the resulting text into something like https://mermaid.live/ and voilà.

I like excalidraw for live discussions but if I want to make more detailed or better looking diagrams, I really enjoy these two tools:

- For drag-and-drop/WYSIWYG, I really like DrawIO. They have a web version https://app.diagrams.net/ but I strongly recommend the desktop version https://github.com/jgraph/drawio-desktop/releases/

- For text-as-diagram, I think Mermaid wins this by default since GitHub added markdown support for these: https://mermaid.live/ (This was github's announcement https://github.blog/2022-02-14-include-diagrams-markdown-fil... )

After hours without being able to solve this problem, I decided to make the chart directly in Mermaid's live editor and export it to PNG and SVG. The big problem is that the letters are still too small, the quality of the PNG is very low, and the SVG only works on the web, not with other applications like PowerPoint and Inkscape.

You can export to PNG and other formats in the Mermaid editor https://mermaid.live/

Copy paste code here: https://mermaid.live