delta-rs
materialize
Our great sponsors
delta-rs | materialize | |
---|---|---|
27 | 116 | |
1,771 | 5,543 | |
6.9% | 1.1% | |
9.7 | 10.0 | |
about 8 hours ago | 1 day ago | |
Rust | Rust | |
Apache License 2.0 | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
delta-rs
-
Delta Lake vs. Parquet: A Comparison
I work at Databricks, but am pretty must just an OSS nerd, mainly focusing on Delta Rust recently: https://github.com/delta-io/delta-rs
I did some keyword research and wrote this post cause lots of folks are doing searches for Delta Lake vs Parquet. I'm just trying to share a fair summary of the tradeoffs with folks who are doing this search. It's a popular post and that's why I figured I would share it here.
-
Working with Rust
Seeing a lot of great libraries coming out with python bindings in the data world e.g delta-rs Polars. I see it growing in this space as a C++ alternative
-
Ideas/Suggestions around setting up a data pipeline from scratch
If I’m not misunderstanding, you could both decode the gRPC protobuf AND write to delta lake in Rust. Tonic, Delta-rs.
-
Polars query engine 0.29.0 released
I know someone will be adding this on the python side in the coming weeks. On the rust side you can use delta-rs with polars. Though you would be compiling both arrow2 and arrow-rs, so that's quite heavy.
-
Delta Lake without Databricks?
You don’t need DBX to use Delta Lake. You can use S3 as the backend and just use the Python Delta Lake library. It works great! https://github.com/delta-io/delta-rs
-
Seeking Recommendations for a Master Data Management Tool
Maybe if I get some free time soon I can formalize into a working example. Been wanting an excuse to try similar concept in delta-rs and polars/duckdb vs databricks/spark vs iceberg/polars.
-
How to write Python extensions in Rust with PyO3
PyO3 is being used to expose the Python bindings to the delta-rs project: https://github.com/delta-io/delta-rs
It's a great way to expose Python bindings because it "feels" Pythonic. Most users run pip install deltalake and are completely unaware that the backend is implemented in Rust.
This is quite a different user experience than Python bindings for Java backends exposed via py4j. The py4j interfaces have the Java feel and require Java to be installed, which most Python users don't like.
- Delta without using Spark
-
Spark open source community is awesome
Yea, there are tons of employees from companies that have made massive contributions to the Spark ecosystem. Apple built Delta Lake with Databricks, see this video for more detail. Lots of Spark PMCs are from various companies. delta-rs was initially built by Scribd and is now actively maintained by engineers at Voltron & other companies. It's awesome the community has so many contributors from various sources.
-
Snowpark equivalent on Databricks?
Have a look at this https://delta-io.github.io/delta-rs/python/
materialize
-
Choosing Between a Streaming Database and a Stream Processing Framework in Python
To fully leverage the data is the new oil concept, companies require a special database designed to manage vast amounts of data instantly. This need has led to different database forms, including NoSQL databases, vector databases, time-series databases, graph databases, in-memory databases, and in-memory data grids. Recent years have seen the rise of cloud-based streaming databases such as RisingWave, Materialize, DeltaStream, and TimePlus. While they each have distinct commercial and technical approaches, their overarching goal remains consistent: to offer users cloud-based streaming database services.
-
Proton, a fast and lightweight alternative to Apache Flink
> Materialize no longer provide the latest code as an open-source software that you can download and try. It turned from a single binary design to cloud-only micro-service
Materialize CTO here. Just wanted to clarify that Materialize has always been source available, not OSS. Since our initial release in 2020, we've been licensed under the Business Source License (BSL), like MariaDB and CockroachDB. Under the BSL, each release does eventually transition to Apache 2.0, four years after its initial release.
Our core codebase is absolutely still publicly available on GitHub [0], and our developer guide for building and running Materialize on your own machine is still public [1].
It is true that we substantially rearchitected Materialize in 2022 to be more "cloud-native". Our new cloud offering offers horizontal scalability and fault tolerance—our two most requested features in the single-binary days. I wouldn't call the new architecture a microservices design though! There are only 2-3 services, each quite substantial, in the new architecture (loosely: a compute service, an orchestration service, and, soon, a load balancing service).
We do push folks to sign up for a free trial of our hosted cloud offering [2] these days, rather than trying to start off by running things locally, as we generally want folks' first impression of Materialize to be of the version that we support for production use cases. A all-in-one single machine Docker image does still exist, if you know where to look, but it's very much use-at-your-own-risk, and we don't recommend using it for anything serious, but it's there to support e.g. academic work that wants to evaluate Materialize's capabilities to incrementally maintain recursive SQL queries.
If folks have questions about Materialize, we've got a lively community Slack [3] where you can connect directly with our product and engineering teams.
[0]: https://github.com/MaterializeInc/materialize/tree/main
-
What I Talk About When I Talk About Query Optimizer (Part 1): IR Design
> the Query Graph Model (QGM) representation is quite abstract and hardcodes many properties, making it exceptionally difficult to understand. Its claimed extensibility is also questionable.
I don't know much about the context, but it was interesting to note that Materialize scrapped their QGM code last year: https://github.com/MaterializeInc/materialize/pull/17139
Also, a couple of interesting projects in the IR space:
- https://substrait.io/ is a cross-language serialization for Relational Algebra
- https://www.lingo-db.com/ is an MLIR-based query engine described extensively in this paper https://db.in.tum.de/~jungmair/papers/p2485-jungmair.pdf?lan...
-
We Built a Streaming SQL Engine
Some recent solutions to this problem include Differential Dataflow and Materialize. It would be neat if postgres adopted something similar for live-updating materialized views.
-
Ask HN: Who is hiring? (October 2023)
Materialize | Full-Time | NYC Office or Remote | https://materialize.com
Materialize is an Operational Data Warehouse: A cloud data warehouse with streaming internals, built for work that needs action on what’s happening right now. Keep the familiar SQL, keep the proven architecture of cloud warehouses but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.
Materialize is the operational data warehouse built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.
Senior/Staff Product Manager - https://grnh.se/69754ebf4us
Senior Frontend Engineer - https://grnh.se/7010bdb64us
===
Investors include Redpoint, Lightspeed and Kleiner Perkins.
-
Ask HN: Who is hiring? (June 2023)
Materialize | EM (Compute), Senior PM | New York, New York | https://materialize.com/
You shouldn't have to throw away the database to build with fast-changing data. Keep the familiar SQL, keep the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date.
That is Materialize, the only true SQL streaming database built from the ground up to meet the needs of modern data products: Fresh, Correct, Scalable — all in a familiar SQL UI.
Engineering Manager, Compute - https://grnh.se/4e14099f4us
Senior Product Manager - https://grnh.se/587c36804us
VP of Marketing - https://grnh.se/9caac4b04us
- What are your favorite tools or components in the Kafka ecosystem?
- Ask HN: Who is hiring? (May 2023)
What are some alternatives?
ClickHouse - ClickHouse® is a free analytics DBMS for big data
risingwave - Scalable Postgres for stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.
delta - An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
openpilot - openpilot is an open source driver assistance system. openpilot performs the functions of Automated Lane Centering and Adaptive Cruise Control for 250+ supported car makes and models.
rust-kafka-101 - Getting started with Rust and Kafka
dbt-expectations - Port(ish) of Great Expectations to dbt test macros
scryer-prolog - A modern Prolog implementation written mostly in Rust.
roapi - Create full-fledged APIs for slowly moving datasets without writing a single line of code.
readyset - Readyset is a MySQL and Postgres wire-compatible caching layer that sits in front of existing databases to speed up queries and horizontally scale read throughput. Under the hood, ReadySet caches the results of cached select statements and incrementally updates these results over time as the underlying data changes.
kubesql - Experimental tool to query K8s API using plain SQL
ballista - Distributed compute platform implemented in Rust, and powered by Apache Arrow.