OpenMLDB VS libpmemobj-cpp

b. Pull the testing code

OpenMLDB has introduced a new service module called OpenM(ysq)LDB, expanding its capabilities to integrate with MySQL infrastructure. This extension redefines the “ML” in OpenMLDB to signify both Machine Learning and MySQL compatibility. Through OpenM(ysq)LDB, users gain the ability to utilize MySQL command-line clients or MySQL SDKs in various programming languages, enabling seamless access to OpenMLDB’s unique online and offline feature calculation capabilities.

OpenMLDB is an open-source, distributed in-memory database system designed for time-series data. It focuses on high performance, reliability, and scalability, making it suitable for handling massive time-series data and real-time computation of online features. In the wave of big data and machine learning, OpenMLDB has emerged as a promising player in the open-source database field, thanks to its powerful data processing capabilities and efficient support for machine learning.

For detailed release notes, please refer to: https://github.com/4paradigm/OpenMLDB/releases/tag/v0.8.4 Feel free to try it out, and discuss it in the official Slack channel (https://join.slack.com/t/openmldb/shared_invite/zt-ozu3llie-K~hn9Ss1GZcFW2~K_L5sMg) if you have any thoughts on improvements or questions!

New to OpenMLDB? Check out the quick workflow and quickstart blog post!

Website: https://openmldb.ai/

Another comment already mentions this, but I guess a functional data structure would be easiest to maintain, even though there's stuff like this for PMEM such that any data structure would work essentially: https://pmem.io/

This[1] Open Source data store I'm maintaining is storing a huge tree of tries eventually on durable storage, but it's essentially a huge persistent index stored in an append-only file. The last layer of "indirect index pages" stores list of pointers to page fragments, which store the actual records (the list however is bound to a predefinded size). A single read-write trx per resource syncs the new page fragments, which mainly only include the updated or new records plus the paths to the root page to a durable device during a postorder traversal of the in-memory log. The last thing is an atomic update, which sets a new offset to point to the new UberPage, the root of the storage. All pages are only word aligned except for RevisionRootPages, which are aligned to a multiple of 256 bytes. All pages are compressed and might in the future optionally be encrypted.

I guess these byte addressable persistent memory devices are an ideal fit for small sized parallel random reads of potentially tiny page fragments to reconstruct a full page in-memory.

However, I hope they'll eventually achieve a better price in comparison to DRAM and SSDs than Intel Optane DC PMEM current price range.

[1] https://github.com/sirixdb/sirix

Yes it is persistent memory programming. https://pmem.io/

Predating Intel's persistent memory a.k.a. NVDIMM, I see a research product "Hotpot" on its practical application as distributed shared memory ( paper and source code ), despite the hardware adoption issue and its apparent research nature (stuck with customized 3.x Linux kernel), I find the core idea not so bound to such a whole new technology stack, covering from low level device driver to application programming toolkit, even depending on sth like 40Gbps RDMA with InfiniBand SAN connection.

It's more work but it looks like you can get the C++ bindings off their github and use them directly. You may have to experiment with different releases to find one compatible with whatever version of the base library you have installed.