Are You Sure You Want to Use MMAP in Your DBMS?

• redb

  24 2,895 9.3 Rust

  An embedded key-value database in pure Rust

I wrote redb (https://github.com/cberner/redb) using mmap, initially. However, I later removed it and switched to read()/write() with my own user space cache. I'm sure it's not as good as the OS page cache, but the difference was only 1.2-1.5x performance on the benchmarks I cared about, and the cache is less than 500 lines of code.

• vmcache

  1 46 5.9 C++

> buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management.

If you don't know about it already, you should look at Viktor Leis' et al's work on this -- kernel module + supporting buffer mgmt code to do the kind of thing you're talking about. The code is not production quality, but the idea is nifty:

https://github.com/viktorleis/vmcache

https://github.com/tuhhosg/exmap

https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up...

I've personally experimented a bit with userfaultfd and this kind of thing, but it's got issues.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• exmap

  1 26 5.7 C
  

> buffers in user space is really immature and needs some better performing API so that we have something in between mmap and buffer management.

If you don't know about it already, you should look at Viktor Leis' et al's work on this -- kernel module + supporting buffer mgmt code to do the kind of thing you're talking about. The code is not production quality, but the idea is nifty:

https://github.com/viktorleis/vmcache

https://github.com/tuhhosg/exmap

https://www.cs.cit.tum.de/fileadmin/w00cfj/dis/_my_direct_up...

I've personally experimented a bit with userfaultfd and this kind of thing, but it's got issues.

Suggest a related project