Chronicle Map VS H2

We’re using JVector for the vector index and Chronicle Map for the article data. There are several things I don’t love about Chronicle Map, but nothing else touches it for simple disk-based key/value performance.

I decided to start with an overview of what open-source options are currently available. When it comes to the implementation of the on-heap cache mechanism, the options are numerous – there is well known: guava, ehcache, caffeine and many other solutions. However, when I began researching cache mechanisms offering the possibility of storing data outside GC control, I found out that there are very few solutions left. Out of the popular ones, only Terracotta is supported. It seems that this is a very niche solution and we do not have many options to choose from. In terms of less-known projects, I came across Chronicle-Map, MapDB and OHC. I chose the last one because it was created as part of the Cassandra project, which I had some experience with and was curious about how this component worked:

Chronicle-Map: Chronicle Map is an in-memory, key-value store, designed for low-latency, and/or multi-process applications.

I've wrangled data sets in the ~600gb range using nothing but plain old Java and a few beefy boxes. This can all be kept in memory, but you have to go off-heap. You can use Chronicle Map and Chronicle Values to model this data and work with it off-heap in a way that's still very clean and object oriented. 128gb of RAM is cheap these days, whether you're in the cloud or not.

The URL should point to a particular comment, but HN removes fragments: https://github.com/h2database/h2database/issues/3686#issueco...

This sounds very much like the idiocy of "infosec" lunkheads who know nothing about what they're "fixing" but if an automated system tells them a CVE exists, they've absolutely got to have it "patched". They don't look into what the claims of the CVE are, or whether their specific use case is vulnerable. They don't know, they don't care, they're not even programmers. All they know is a box needs ticking.

A similar thing happened with h2database - a "security researcher" found that if you do something you're told not to do, then bad things happen.. but they demanded and got a CVE allocated anyway. Anyone who looks at it realises it's bullshit, but the mere existence of a CVE is all that matters to these idiots.

What the h2database developer said about it: https://github.com/h2database/h2database/issues/3686#issueco...

> I struggle to understand why I should feel the slightest shred of sympathy for "major corporations" that are using a volunteer-developed open-source project. Feel free to get your corporation to pay someone to deal with this, or pay for a similar commercial library.

H2 - Free, Embedded & Open source

H2 2.1

H2 for mocking relational database connections

It's open-source and written in Java, so you can even create custom procedures and register them straight in your application!

Firstly, let's set up a basic Spring Boot application. We can use the Spring Initializr with Spring Data JPA, H2 Database, Lombok added. H2 Database will provide us with a simple database, and Spring Data JPA will allow us to easily interact with it using Hibernate. Lombok will make it easier to write concise and readable classes.

Sorting chunks by similarity: commonly used tools don't do that. Most archive tools only sort by file type.

I wrote a tool that chunks the data (into variable-sized blocks, to re-sync if there are multiple files that have different length prefixes, but that's another story), and then sorts the chunks by LSH (locality sensitive hash). LSH is used by search engines to detect similar text. It can compress directories that contain multiple version of e.g. source code very well (e.g. trunk, branches). https://github.com/h2database/h2database/blob/master/h2/src/...

I discussed this approach with a researcher in this area in January 2020. AFAIK there is active research in this area, specially to compress DNA sequences. But he also wasn't aware of papers or research in this area for general-purpose data compression.

So, I think this area is largely uncharted. I would be interested (as a hobby side project) to help, if somebody is interested.