Our great sponsors
-
delta
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs (by delta-io)
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
>the delta also still keeps partitioning information in the hive metastore, while iceberg keeps it in storage, making it a far superior design.
Check out https://github.com/delta-io/delta/blob/3ffb30d86c6acda9b59b9... when you get a chance. You don't need hive metastore to query delta tables since all metadata for a Delta table is stored alongside the data
>they did not include features like optimizing small files
Related posts
- [D] Is there other better data format for LLM to generate structured data?
- Delta vs Iceberg: make love not war
- Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML
- Medallion/lakehouse architecture data modelling
- whenNotMatchedBySourceUpdate not existing? Trying to upsert parquet into Delta table