Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 20 Rust Kafka Projects
-
risingwave
Cloud-native SQL stream processing, analytics, and management. KsqlDB and Apache Flink alternative. 🚀 10x more productive. 🚀 10x more cost-efficient.
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
Ockam
Orchestrate end-to-end encryption, cryptographic identities, mutual authentication, and authorization policies between distributed applications – at massive scale.
-
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
-
-
-
-
-
-
rust-kafka-producer-partitioner-example
Example Code for "How to Build a Kafka Producer in Rust with Partitioning"
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: Proton, a fast and lightweight alternative to Apache Flink | news.ycombinator.com | 2024-01-30How does this compare to RisingWave and Materialize?
https://github.com/risingwavelabs/risingwave
Project mention: Ask HN: How Can I Make My Front End React to Database Changes in Real-Time? | news.ycombinator.com | 2024-04-17[2] https://materialize.com/
disclosure: I work at Ockam.
The Portals for Mac app is an example of the type of thing you could build using the open source stack of protocols. The README (linked by parent) links out to all of the relevant parts of the protocol documentation to explain how these work together. The NAT Traversal (https://github.com/build-trust/ockam/blob/develop/examples/a...) part of the README is probably the best explanation of why the free relay you get via Ockam Orchestrator is a useful part of this demo.
As for why would anyone trust this: The protocols are designed so you absolutely don't have to trust the relay. Trust is pushed out to the edges that you control and so you're not susceptible to a MITM attack if something like a relay is compromised. The protocol design for all of this is open and documented, and was independently audited by (IMO) some of the best in the business, Trail of Bits: https://docs.ockam.io/reference/protocols.
Project mention: Show HN: Hatchet – Open-source distributed task queue | news.ycombinator.com | 2024-03-08That's exactly why we built Svix[1]. Building webhooks services, even with amazing tools like FastAPI, Celery and Redis is still a big pain. So we just built a product to solve it.
Hatchet looks cool nonetheless. Queues are a pain for many other use-cases too.
1: https://www.svix.com
Project mention: Rust Cpp Interop via Cxx, Autocxx / any best practices out there | /r/rust | 2023-06-19I use this library a lot and it's got some nice touches for how to handle wrapping a C library: https://github.com/fede1024/rust-rdkafka
Project mention: Vector: a Rust-based lightweight alternative to Fluentd/Logstash | news.ycombinator.com | 2023-09-26
Rust can offer incredible cost savings when you can use it in place of spark to interact with your delta lake. One such project was kafka-delta-ingest. The developers were able to reduce the cost of running the pipeline by over 90%. However, most of this stuff is still very experimental and not ready for production but you will definitely be seeing more projects like this just based on how much money can be saved.
Project mention: Ask HN: What apps have you created for your own use? | news.ycombinator.com | 2023-12-12I wrote pq [1] (protobuf parser cli) at a company where I was told to "just use the tool another engineer wrote" which was in C++, in a really uncompileable/abandoned/unusable state
I wrote goat [2] (EBS disk attacher) at the same company on a solo project where I needed to create a "Kafka-cluster-IaC" recipe in Terraform and wanted us to be able to replace EC2 broker instances dynamically but preserve their data on the EBS volume
[1] https://github.com/sevagh/pq
[2] https://github.com/sevagh/goat
Project mention: My first published crate, Omniqueue: a queue abstraction layer for Rust | /r/rust | 2023-06-08
Project mention: Show HN: Kiwi – End-to-End Kafka Subscriptions with WebAssembly | news.ycombinator.com | 2024-04-06
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Rust Kafka related posts
- Show HN: Kiwi – End-to-End Kafka Subscriptions with WebAssembly
- Arryo 0.8 released — streaming SQL engine
- We Built a Streaming SQL Engine
- RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
- Using rust for DE activities?
- Arroyo
- My first published crate, Omniqueue: a queue abstraction layer for Rust
-
A note from our sponsor - InfluxDB
www.influxdata.com | 26 Apr 2024
Index
What are some of the best open-source Kafka projects in Rust? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | risingwave | 6,283 |
| 2 | materialize | 5,567 |
| 3 | Ockam | 4,347 |
| 4 | paradedb | 3,803 |
| 5 | arroyo | 3,275 |
| 6 | svix-webhooks | 2,075 |
| 7 | rust-rdkafka | 1,477 |
| 8 | kafka-rust | 1,144 |
| 9 | flowgger | 801 |
| 10 | kafka-delta-ingest | 323 |
| 11 | oura | 253 |
| 12 | graphql-rust-demo | 186 |
| 13 | pq | 167 |
| 14 | shotover-proxy | 79 |
| 15 | omniqueue-rs | 50 |
| 16 | kiwi | 46 |
| 17 | rust-kafka-101 | 3 |
| 18 | kafka-tracing-blog-post-example-code | 2 |
| 19 | kafka-rust-demo | 2 |
| 20 | rust-kafka-producer-partitioner-example | 1 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com