Building a new vector based storage model

• QuestDB

  311 13,501 9.7 Java

  An open source time-series database for fast ingest and SQL queries

  

Hey everyone! I'm Vlad, co-founder/CTO of QuestDB, an open-source database for time series data focusing on performance (https://github.com/questdb/questdb).

We launched QuestDB last summer [1] and this was our first iteration. We also showed some of our querying capabilities by putting QuestDB on the internet with a 1.6 billion rows data set to query, live [2]! Our storage model is vector-based and append-only. This meant that all incoming data had to arrive in the correct time order. This worked well for some use cases but we quickly realised its limitation in "real-world" use cases, where data doesn't always land at the database in chronological order (we explain this in more details). In short, we were not ready to face the world! We were naive and had a lot to learn. We saw plenty of developers and users come and go specifically because of this technical limitation.

After we decided to deal with out-of-order data, the big decision to make was which direction we would choose to tackle the problem. LSM trees seemed an obvious choice, but we chose an alternative route so we could keep the performance we spent years building. Our latest release supports out-of-order ingestion by re-ordering data on the fly, and we're happy to share our approach to solving this challenge.

Additionally, we had many people asking about the differences between QuestDB and other open-source databases and why users should consider giving it a try instead of other systems. When we launched on HN, we saw even more interest in technical analyses, and readers showed a lot of interest in side-by-side comparisons to other databases on the market. One suggestion [3] that we thought would be great to try out was to benchmark ingestion and query speeds using the Time Series Benchmark Suite (TSBS) [4] developed by TimescaleDB. We're super excited to share the results in the article.

I'd love to hear your feedback on this release and I'm happy to answer any questions you have, along with the rest of the QuestDB team.

Vlad

[1] https://news.ycombinator.com/item?id=23975807

• tsbs

  76 1,216 1.6 Go

  Time Series Benchmark Suite, a tool for comparing and evaluating databases for time series data

  

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project