Apache Calcite VS ANTLR

> Make diff work on more than just SQLite.

Another way of doing this that I've been wanting to do for a while is to implement the DIFF operator in Apache Calcite[0]. Using Calcite, DIFF could be implemented as rewrite rules to generate the appropriate SQL to be directly executed against the database or the DIFF operator can be implemented outside of the database (which the original paper shows is more efficient).

[0] https://calcite.apache.org/

Yes, planetiler rocks and the memory mapped collections enabled us to remove our dependency to rocksdb.

From my perspective, planetiler started as an effort to generate vector tiles from the OpenMapTile schema as fast as possible (pbf -> mvt). By contrast, Baremaps started as an effort to create a new schema and style from the ground up. In this regard, having a database (pbf -> db <- mvt) enables to live reload changes made in the configuration files. The database has a cost, but also comes with additional advantages (updates, dynamic data, generation of tiles at zoom levels 16+, etc.).

That being said, I think the two projects overlap and I hope we will find opportunities to collaborate in the future. For instance, whereas PostgreSQL is still required in Baremaps, I recently ported a lot of the ST_ function of Postgis to Apache Calcite with the intent to execute SQL on fast memory mapped collection.

https://github.com/apache/calcite/blob/main/core/src/main/ja...

A planet wide import in Postgis currently takes about 4 hours with the COPY API (easy to parallelize) followed by about 12 hours of simplification in Postgis (not easy to parallelize). I will try to publish a detailed benchmark in the future.

Use a SQL Parser like sqlglot or Apache Calcite to compile user's query into an AST.

The SQL Federation engine contains processes such as SQL Parser, SQL Binder, SQL Optimizer, Data Fetcher and Operator Calculator, suitable for dealing with co-related queries and subqueries cross multiple database instances. At the underlying layer, it uses Calcite to implement RBO (Rule Based Optimizer) and CBO (Cost Based Optimizer) based on relational algebra, and query the results through the optimal execution plan.

Awesome to see work in the DB wire compatible space. On the MySQL side, there was MySQL Proxy (https://github.com/mysql/mysql-proxy), which was scriptable with Lua, with which you could create your own MySQL wire compatible connections. Unfortunately it appears to have been abandoned by Oracle and IIRC doesn't work with 5.7 and beyond. I used it in the past to hack together a MySQL wire adapter for Interana (https://scuba.io/).

I guess these days the best approach for connecting arbitrary data sources to existing drivers, at least for OLAP, is Apache Calcite (https://calcite.apache.org/). Unfortunately that feels a little more involved.

For anyone interested, Apache Calcite[0] is an open source data management framework which seems to do many of the same things that Hydra claims to do, but taking a different approach. Operating as a Java library, Calcite contains "adapters" to many different data sources from existing JDBC connectors to Elasticsearch to Cassandra. All of these different data sources can be joined together as desired. Calcite also has it's own optimizer which is able to push down relevant parts of the query to the different data sources. However, you get full SQL on data sources which don't support it, with Calcite executing the remaining bits itself.

Unfortunately, I would not be too surprised if Calcite was found to be less performance-optimized than Hydra. That said, there are users of Calcite at Google, Uber, Spotify, and others who have made great use of various parts of the framework.

[0] https://calcite.apache.org/

Abstraction Layer - You can use something like Calcite to abstract out your data storage. https://calcite.apache.org/

antlr https://github.com/antlr/antlr4

Download the ANTLR jar from https://www.antlr.org/download/antlr-4.9.2-complete.jar Add the ANTLR jar to your project's classpath. Install the ANTLR Kotlin target by following the instructions at https://github.com/antlr/antlr4/blob/master/doc/targets/Kotlin.md Next, you'll need a Perl grammar file for ANTLR:

Write a context-free grammar for it, commonly written in Backus Naur Form, and use that to write a parser. There are tools named "parser generators" like antlr4 that can automatically convert a BNF grammar into a parser.

Tutorial on using it in Java: https://www.baeldung.com/java-antlr Github project itself with docs and examples: https://github.com/antlr/antlr4

Depending on how complex your expressions are, you might consider using something like antlr and writing your own parser for it. Setting up something to handle math and string operations wouldn’t be very hard and then you can control the syntax however you like. You can use a visitor and visit each node in the syntax tree and return the result of each sub-expression.

> ...than it would be to learn the exact syntax and quirks and possibly bugs of someone else's implementation...

Yup. Also, having deep knowledge of the language is required.

SQLite's grammar is neat. Creating a compatible parser would make a fun project. Here's a pretty good example: https://github.com/bkiers/sqlite-parser (Actual ANTLR 4 grammar: https://github.com/bkiers/sqlite-parser/blob/master/src/main... )

Postgres, which tries to be compliant with the latest standards, however...

SQL-2016 is a beast. Not to mention all the dialects.

I'm updating my personal (soon to be FOSS) grammar from ANTLR 3 LL(k) to ANTLR 4 ALL().

I've long had a working knowledge of SQL-92, with some SQL-1999 (eg common table expressions).

But the new structures and extensions are a bit overwhelming.

Fortunately, ANTLR project has ~dozen FOSS grammars to learn from. https://github.com/antlr/grammars-v4/tree/master/sql

They mostly mechanically translate BNFs to LL(k) with some ALL(). Meaning few take advantage of left-recursion. https://github.com/antlr/antlr4/blob/master/doc/left-recursi...

Honestly, I struggled to understand these grammars. Plus, not being conversant with the SQL-2016 was a huge impediment. Just finding a succinct corbis of test cases was a huge hurdle for me.

Fortunately, the H2 Database project is a great resource. https://github.com/h2database/h2database/tree/master/h2/src/...

Now for the exciting conclusion...

My ANTLR grammar which passes all of H2's tests looks nothing like any of the official or product specific BNFs.

Further, I found discrepancy between the product specific BNFs and their implementations.

So a lot of trial & error is required for a "real world" parser. Which would explain why the professional SQL parsing tools charge money.

I still think creating a parser for SQLite is a great project.

can be used to parse

I already posted here about a project, but I could also use help on Mantle. It's a new command framework powered by ANTLR, if that's something you're interested in.

ive been tryng to work with antlr4 and go but it seems that i cant import the runtime, it says that the antlr runtime isnt in the gopath but ive already done go get github.com/antlr/antlr4/runtime/antlr4 and i dont know what to do now, im on windows if anyone knows what to do it would be very helpful. thanks already