H2 VS MapDB

The URL should point to a particular comment, but HN removes fragments: https://github.com/h2database/h2database/issues/3686#issueco...

This sounds very much like the idiocy of "infosec" lunkheads who know nothing about what they're "fixing" but if an automated system tells them a CVE exists, they've absolutely got to have it "patched". They don't look into what the claims of the CVE are, or whether their specific use case is vulnerable. They don't know, they don't care, they're not even programmers. All they know is a box needs ticking.

A similar thing happened with h2database - a "security researcher" found that if you do something you're told not to do, then bad things happen.. but they demanded and got a CVE allocated anyway. Anyone who looks at it realises it's bullshit, but the mere existence of a CVE is all that matters to these idiots.

What the h2database developer said about it: https://github.com/h2database/h2database/issues/3686#issueco...

> I struggle to understand why I should feel the slightest shred of sympathy for "major corporations" that are using a volunteer-developed open-source project. Feel free to get your corporation to pay someone to deal with this, or pay for a similar commercial library.

> ...than it would be to learn the exact syntax and quirks and possibly bugs of someone else's implementation...

Yup. Also, having deep knowledge of the language is required.

SQLite's grammar is neat. Creating a compatible parser would make a fun project. Here's a pretty good example: https://github.com/bkiers/sqlite-parser (Actual ANTLR 4 grammar: https://github.com/bkiers/sqlite-parser/blob/master/src/main... )

Postgres, which tries to be compliant with the latest standards, however...

SQL-2016 is a beast. Not to mention all the dialects.

I'm updating my personal (soon to be FOSS) grammar from ANTLR 3 LL(k) to ANTLR 4 ALL().

I've long had a working knowledge of SQL-92, with some SQL-1999 (eg common table expressions).

But the new structures and extensions are a bit overwhelming.

Fortunately, ANTLR project has ~dozen FOSS grammars to learn from. https://github.com/antlr/grammars-v4/tree/master/sql

They mostly mechanically translate BNFs to LL(k) with some ALL(). Meaning few take advantage of left-recursion. https://github.com/antlr/antlr4/blob/master/doc/left-recursi...

Honestly, I struggled to understand these grammars. Plus, not being conversant with the SQL-2016 was a huge impediment. Just finding a succinct corbis of test cases was a huge hurdle for me.

Fortunately, the H2 Database project is a great resource. https://github.com/h2database/h2database/tree/master/h2/src/...

Now for the exciting conclusion...

My ANTLR grammar which passes all of H2's tests looks nothing like any of the official or product specific BNFs.

Further, I found discrepancy between the product specific BNFs and their implementations.

So a lot of trial & error is required for a "real world" parser. Which would explain why the professional SQL parsing tools charge money.

I still think creating a parser for SQLite is a great project.

H2 - Free, Embedded & Open source

H2 2.1

H2 for mocking relational database connections

It's open-source and written in Java, so you can even create custom procedures and register them straight in your application!

Firstly, let's set up a basic Spring Boot application. We can use the Spring Initializr with Spring Data JPA, H2 Database, Lombok added. H2 Database will provide us with a simple database, and Spring Data JPA will allow us to easily interact with it using Hibernate. Lombok will make it easier to write concise and readable classes.

Sorting chunks by similarity: commonly used tools don't do that. Most archive tools only sort by file type.

I wrote a tool that chunks the data (into variable-sized blocks, to re-sync if there are multiple files that have different length prefixes, but that's another story), and then sorts the chunks by LSH (locality sensitive hash). LSH is used by search engines to detect similar text. It can compress directories that contain multiple version of e.g. source code very well (e.g. trunk, branches). https://github.com/h2database/h2database/blob/master/h2/src/...

I discussed this approach with a researcher in this area in January 2020. AFAIK there is active research in this area, specially to compress DNA sequences. But he also wasn't aware of papers or research in this area for general-purpose data compression.

So, I think this area is largely uncharted. I would be interested (as a hobby side project) to help, if somebody is interested.

I decided to start with an overview of what open-source options are currently available. When it comes to the implementation of the on-heap cache mechanism, the options are numerous – there is well known: guava, ehcache, caffeine and many other solutions. However, when I began researching cache mechanisms offering the possibility of storing data outside GC control, I found out that there are very few solutions left. Out of the popular ones, only Terracotta is supported. It seems that this is a very niche solution and we do not have many options to choose from. In terms of less-known projects, I came across Chronicle-Map, MapDB and OHC. I chose the last one because it was created as part of the Cassandra project, which I had some experience with and was curious about how this component worked:

So, it's an object database, like Zope's ZODB on Python?

I like the idea, but I'd like to learn about use cases for it.

Otherwise, in Java, MapDB is about as far as I'd be willing to go: https://github.com/jankotek/mapdb/

Anyway, without further ado, I found MapDB (https://github.com/jankotek/mapdb) which does exactly that. Of course, they also provide their own Java collection implementations as well, so I suspect using it with Vavr would be a poor idea, but it is very cool in its own right anyway. Of course, there is also Apache Derby and HSQLDB, and those great options with a long history as well. I haven't played with these in a while though, so I might give them a try again soon for some personal stuff.

In Java I like

https://mapdb.org/

It is more feature rich than you want but in Python I'd probably just use sqlite3 since it is in the standard library.

I have had good results with mapdb