-
delta
An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs (by delta-io)
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
I’d suggest looking at the open table formats. Delta lake does an excellent job at providing batch and streaming APIs for Spark. This would unify your workloads. It would follow the medallion architecture which is a bit more popular lately. Aspects of the lamda architecture can still be present in the medallion model, especially when real-time requirements are present.
Related posts
-
[D] Is there other better data format for LLM to generate structured data?
-
Delta vs Iceberg: make love not war
-
Databricks Strikes $1.3B Deal for Generative AI Startup MosaicML
-
Medallion/lakehouse architecture data modelling
-
whenNotMatchedBySourceUpdate not existing? Trying to upsert parquet into Delta table