Inserting 1.1M rows/s from Pandas into QuestDB with Arrow, Rust & Cython

• py-tsbs-benchmark

  11 28 1.1 Python

  Benchmark ingestion of the TSBS "dev ops" dataset into QuestDB via ILP using the `questdb` Python library and Pandas.

  

See latest commit: https://github.com/questdb/py-tsbs-benchmark/commit/22ef9fcd61caf8115046b19a9d610bc838b50adf

• c-questdb-client

  2 39 6.6 C++

  Rust, C and C++ client for QuestDB InfluxDB Line Protocol

  

The main code is in Cython (and the fact that one can inspect the generated C is pretty neat) with auxilliary code in Rust. The main serialization logic is in Rust and it's in a separate repo: https://github.com/questdb/c-questdb-client/tree/main/questdb-rs.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Numba

  124 9,432 9.9 Python

  NumPy aware dynamic Python compiler using LLVM

  

You can also check out the Numba Python package that can sometimes help with some things: https://numba.pydata.org/. For example, I use it to implement a Numpy ufunc in the benchmark to generate a bounded random walk which would would be quite slow to do otherwise in pure Python for 10M rows: https://github.com/questdb/py-tsbs-benchmark/blob/ff0885c4aef852ca9a9f89919d0ad36bd6faec8f/py_tsbs_benchmark/bench_pandas.py#L15

Suggest a related project