|5 days ago||5 days ago|
|Apache License 2.0||Apache License 2.0|
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Go devs that learned Rust, what are your thoughts on it?
7 projects | reddit.com/r/golang | 8 Jan 2023
Apache Pegasus – A a distributed key-value storage system
4 projects | news.ycombinator.com | 6 Oct 2022
TiKV is basically a layer on top of rocksdb https://github.com/tikv/tikv/blob/956610725039835557e7516828...
Surrealdb – FOSS document-graph database, for the realtime web in Rust
6 projects | news.ycombinator.com | 16 Sep 2022
> Many,many smart people…
If you look inside the code you can see the stated features are a result of underlying engine (TiKV  also in c and rust from pingcap). Surrealdb is standing on shoulders of giants at present, they are TiKV, FoundationDB and rocksdb. The feature set they mentioned mostly coming from TiKV at present.
Cloud database for tomorrow's applications (written in Rust)
7 projects | news.ycombinator.com | 22 Aug 2022
Hi Diggsey, great question. We are currently focussed on functionality and stability, and then will draw our attention to performance. Coming this week we have a RocksDB storage implementation. We've only just launched our initial beta version, and we know there is a lot of improvement and work to be done (some of these performance issues we know about already and are on our Github issues list).
With regards to the consistency/isolation model, SurrealDB sits on top of a number of key-value stores. By using the distributed highly-available TiKV storage backend, https://tikv.org, (and we have a FoundationDB integration in the works), the database is designed to be highly-scalable and highly-available. The same guarantees (albeit just single-node, so no high-availability or scalability) will be available with the RocksDB implementation coming this week. By sitting on top of these key-value stores, SurrealDB ensures that all transactions are ACID compliant. We don't want to go for speed (for instance by writing to /dev/null) over anything, but want SurrealDB to be a reliable and performant backend for any application. Obviously we have a way to go to catch up with PostgreSQL (launched in 1996), but we will strive to get there!
CeresDB: A high-performance, distributed, schema-less and time-series database
3 projects | news.ycombinator.com | 15 Jun 2022
If you are looking for a production ready distributed store written in Rust. Check out TiKV(https://github.com/tikv/tikv), which was also mentioned in the acknowledge section of the project's README.
There's also a full-featured distributed RDBMS called TiDB built on top of TiKV.
Fly.io – Free Postgres Databases (and free storage volumes, up to 3GB total)
6 projects | news.ycombinator.com | 20 Jan 2022
Fair enough. Indeed I didn't consider support costs. Thank you for your answer!
Actually let me ask another thing. Your FAQ mentions you're considering hosting CockroachDB as a drop-in distributed replacement for PostgreSQL , and also you currently offer a distributed, eventually consistent PostgreSQL replication solution .
Is either Tikv  (distributed key-value store) or Tidb  (distributed database with a mysql interface, built on top of Tikv) on your radar?
You already offer Redis as a key-value store, but Tikv has an amazing property: it ensures strong consistency globally (not eventual consistency). Tidb, being built on top of Tikv, also has strong consistency.
NoSQL and Key-Value storage systems based on Rust (Redis and Tarantool replacements in Rust)
10 projects | reddit.com/r/rust | 17 Dec 2021
tikv — A distributed KV database in Rust
Belajar Rust 01 - Mengenal Bahasa Pemrograman Rust
6 projects | dev.to | 28 Aug 2021
TiKV: basis data key-value transaksional yang terdistribusi.
Dive Deep into TiKV Transactions: The Life Story of a TiKV Prewrite Request
2 projects | dev.to | 23 Aug 2021
Before I introduce this phase, I'd like to talk about the batch system. It is the cornerstone of TiKV's multi-raft implementation.2 projects | dev.to | 23 Aug 2021
TiKV is a distributed key-value storage engine, which is based on the designs of Google Spanner, F1, and HBase. However, TiKV is much simpler to manage because it does not depend on a distributed file system.
Using Rust to write a Data Pipeline. Thoughts. Musings.
5 projects | reddit.com/r/rust | 14 Jan 2023
Demystifying Apache Arrow
2 projects | news.ycombinator.com | 10 Jan 2023
You can see some of the benchmarks in DataFusion (part of the Arrow project and built with Arrow as the underlying in-memory format) https://github.com/apache/arrow-datafusion/blob/master/bench...
Disclaimer: I'm a committer on the Arrow project and contributor to DataFusion.
Scala or Rust? which one will rule in future?
4 projects | reddit.com/r/dataengineering | 23 Dec 2022
polars and datafusion seem very promising
Welcome to Comprehensive Rust
10 projects | news.ycombinator.com | 22 Dec 2022
Rust has amazing integration with Python through PyO3  so see it like a safe alternative for high performance calculations. The ecosystem itself is starting to come together exciting projects like Polars  (Pandas alternative), nalgebra , Datafusion  and Ballista 
Command-line data analytics made easy
6 projects | news.ycombinator.com | 3 Nov 2022
It could be the NDJSON parser (DF source: ) or could be a variety of other factors. Looking at the ROAPI release archive , it doesn't ship with the definitive `columnq` binary from your comment, so it could also have something to do with compilation-time flags.
FWIW, we use the Parquet format with DataFusion and get very good speeds similar to DuckDB , e.g. 1.5s to run a more complex aggregation query `SELECT date_trunc('month', tpep_pickup_datetime) AS month, COUNT(*) AS total_trips, SUM(total_amount) FROM tripdata GROUP BY 1 ORDER BY 1 ASC)` on a 55M row subset of NY Taxi trip data.6 projects | news.ycombinator.com | 3 Nov 2022
SPyQL is really cool and its design is very smart, with it being able to leverage normal Python functions!
As far as similar tools go, I recommend taking a look at DataFusion, dsq, and OctoSQL.
DataFusion is a very (very very) fast command-line SQL engine but with limited support for data formats.
dsq is based on SQLite which means it has to load data into SQLite first, but then gives you the whole breath of SQLite, it also supports many data formats, but is slower at the same time.
OctoSQL is faster, extensible through plugins, and supports incremental query execution, so you can i.e. calculate a running group by + count while tailing a log file. It also supports normal databases, not just file formats, so you can i.e. join with a Postgres table.
Disclaimer: Author of OctoSQL
Welcome to InfluxDB IOx: InfluxData’s New Storage Engine
5 projects | news.ycombinator.com | 26 Oct 2022
Just wanted to give a shout out to Apache DataFusion that IOx relies on a lot (and contributes to as well!).
It's a framework for writing query engines in Rust that takes care of a lot of heavy lifting around parsing SQL, type casting, constructing and transforming query plans and optimizing them. It's pluggable, making it easy to write custom data sources, optimizer rules, query nodes etc.
It's has very good single-node performance (there's even a way to compile it with SIMD support) and Ballista  extends that to build it into a distributed query engine.
Plenty of other projects use it besides IOx, including VegaFusion, ROAPI, Cube.js's preaggregation store. We're heavily using it to build Seafowl , an analytical database that's optimized for running SQL queries directly from the user's browser (caching, CDNs, low latency, some WASM support, all that fun stuff).
GlueSQL: A SQL database engine written as a library in Rust
4 projects | news.ycombinator.com | 22 Oct 2022
Another "database toolkit" project that I've recently learned about is Apache DataFusion, also written in rust and uses Arrow memory format:
Rust is showing a lot of promise in the DataFrame / tabular data space
9 projects | reddit.com/r/rust | 4 Oct 2022
[arrow-datafusion](https://github.com/apache/arrow-datafusion) is another great DataFrame library, especially if you like running SQL queries. It's so easy to query a Parquet / CSV dataset with SQL using DataFusion. I've run local benchmarks and it's super fast. The DataFusion docs are a bit lacking, which is a shame, for such a developed and amazing library. I hope to make these better and help spread the world about how truly amazing this lib is.
Steampipe – Select * from Cloud;
13 projects | news.ycombinator.com | 30 Sep 2022
To add somewhat of a counterpoint to the other response, I've tried the Steampipe CSV plugin and got 50x slower performance vs OctoSQL, which is itself 5x slower than something like DataFusion. The CSV plugin doesn't contact any external API's so it should be a good benchmark of the plugin architecture, though it might just not be optimized yet.
That said, I don't imagine this ever being a bottleneck for the main use case of Steampipe - in that case I think the APIs themselves will always be the limiting part. But it does - potentially - speak to what you can expect if you'd like to extend your usage of Steampipe to more than just DevOps data.
Disclaimer: author of OctoSQL
What are some alternatives?
redis-rs - Redis library for rust
polars - Fast multi-threaded, hybrid-out-of-core DataFrame library in Rust | Python | Node.js
rust-etcd - An etcd client library for Rust.
ClickHouse - ClickHouse® is a free analytics DBMS for big data
rust-rocksdb - rust wrapper for rocksdb
cassandra-rs - Cassandra (CQL) driver for Rust, using the DataStax C/C++ driver under the covers.
db-benchmark - reproducible benchmark of database-like ops
diesel - A safe, extensible ORM and Query Builder for Rust
rust-postgres - Native PostgreSQL driver for the Rust programming language
KeyDB - A Multithreaded Fork of Redis