c-questdb-client
py-tsbs-benchmark
Our great sponsors
c-questdb-client | py-tsbs-benchmark | |
---|---|---|
2 | 11 | |
39 | 28 | |
- | - | |
6.6 | 1.1 | |
17 days ago | about 1 year ago | |
C++ | Python | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
c-questdb-client
-
Inserting 1.8M Rows/S from Pandas into QuestDB with Arrow, Rust and Cython
Hi, I'm the original author of the QuestDB Python client library and benchmark.
It all started when we had one of our users needing to insert quite a bit of data into our database quickly from Pandas. They had a dataframe that took 25 minutes to serialize row-by-row iterating through the dataframe. The culprit was .iterrows(). Now it's a handful of seconds.
This took a few iterations: At first I thought this could all be handled by Python buffer protocol, but that turned out to create a whole bunch of copies, so for a number of dtypes the code now uses Arrow when it's zero-copy.
The main code is in Cython (and the fact that one can inspect the generated C is pretty neat) with supporting code in Rust. The main serialization logic is in Rust and it's in a separate repo: https://github.com/questdb/c-questdb-client/tree/main/questd....
-
Inserting 1.1M rows/s from Pandas into QuestDB with Arrow, Rust & Cython
The main code is in Cython (and the fact that one can inspect the generated C is pretty neat) with auxilliary code in Rust. The main serialization logic is in Rust and it's in a separate repo: https://github.com/questdb/c-questdb-client/tree/main/questdb-rs.
py-tsbs-benchmark
- Inserting 1.8M Rows/S from Pandas into QuestDB with Arrow, Rust and Cython
-
Inserting 1.1M rows/s from Pandas into QuestDB with Arrow, Rust & Cython
See latest commit: https://github.com/questdb/py-tsbs-benchmark/commit/22ef9fcd61caf8115046b19a9d610bc838b50adf
- GitHub Benchmark: Inserting 1.1M rows / second into QuestDB from Pandas with Arrow, Cython & Rust
What are some alternatives?
Numba - NumPy aware dynamic Python compiler using LLVM
Pandas - Flexible and powerful data analysis / manipulation library for Python, providing labeled data structures similar to R data.frame objects, statistical functions, and much more