ANTLR VS zetasql

antlr https://github.com/antlr/antlr4

Download the ANTLR jar from https://www.antlr.org/download/antlr-4.9.2-complete.jar Add the ANTLR jar to your project's classpath. Install the ANTLR Kotlin target by following the instructions at https://github.com/antlr/antlr4/blob/master/doc/targets/Kotlin.md Next, you'll need a Perl grammar file for ANTLR:

Write a context-free grammar for it, commonly written in Backus Naur Form, and use that to write a parser. There are tools named "parser generators" like antlr4 that can automatically convert a BNF grammar into a parser.

Tutorial on using it in Java: https://www.baeldung.com/java-antlr Github project itself with docs and examples: https://github.com/antlr/antlr4

Depending on how complex your expressions are, you might consider using something like antlr and writing your own parser for it. Setting up something to handle math and string operations wouldn’t be very hard and then you can control the syntax however you like. You can use a visitor and visit each node in the syntax tree and return the result of each sub-expression.

> ...than it would be to learn the exact syntax and quirks and possibly bugs of someone else's implementation...

Yup. Also, having deep knowledge of the language is required.

SQLite's grammar is neat. Creating a compatible parser would make a fun project. Here's a pretty good example: https://github.com/bkiers/sqlite-parser (Actual ANTLR 4 grammar: https://github.com/bkiers/sqlite-parser/blob/master/src/main... )

Postgres, which tries to be compliant with the latest standards, however...

SQL-2016 is a beast. Not to mention all the dialects.

I'm updating my personal (soon to be FOSS) grammar from ANTLR 3 LL(k) to ANTLR 4 ALL().

I've long had a working knowledge of SQL-92, with some SQL-1999 (eg common table expressions).

But the new structures and extensions are a bit overwhelming.

Fortunately, ANTLR project has ~dozen FOSS grammars to learn from. https://github.com/antlr/grammars-v4/tree/master/sql

They mostly mechanically translate BNFs to LL(k) with some ALL(). Meaning few take advantage of left-recursion. https://github.com/antlr/antlr4/blob/master/doc/left-recursi...

Honestly, I struggled to understand these grammars. Plus, not being conversant with the SQL-2016 was a huge impediment. Just finding a succinct corbis of test cases was a huge hurdle for me.

Fortunately, the H2 Database project is a great resource. https://github.com/h2database/h2database/tree/master/h2/src/...

Now for the exciting conclusion...

My ANTLR grammar which passes all of H2's tests looks nothing like any of the official or product specific BNFs.

Further, I found discrepancy between the product specific BNFs and their implementations.

So a lot of trial & error is required for a "real world" parser. Which would explain why the professional SQL parsing tools charge money.

I still think creating a parser for SQLite is a great project.

can be used to parse

I already posted here about a project, but I could also use help on Mantle. It's a new command framework powered by ANTLR, if that's something you're interested in.

ive been tryng to work with antlr4 and go but it seems that i cant import the runtime, it says that the antlr runtime isnt in the gopath but ive already done go get github.com/antlr/antlr4/runtime/antlr4 and i dont know what to do now, im on windows if anyone knows what to do it would be very helpful. thanks already

There are even table-valued functions.

These things are not widespread, and differ by implementation, and the way these are used by clients are copy-and-paste. Something as thoughtful as ZetaSQL https://github.com/google/zetasql does not have mechanisms for structuring (modules, packages, interfaces). SQL will not, cannot evolve into such a direction (or, anything that evolves, will not be recognizable as SQL).

Hi, I develop a BigQuery emulator ( https://github.com/goccy/bigquery-emulator ) from early 2022. It is written in Go, but can be used from bq command line tool and other language's (e.g. Python ) client SDK after installing docker image or released binary. It currently supports over 200 of the nearly 330 standard functions in BigQuery and all data types except GEOGRAPHY ( see https://github.com/goccy/go-zetasqlite#status for details ). ZetaSQL ( https://github.com/google/zetasql ) is used to parse and analyze queries.

We are using a Python client binding for ZetaSQL GRPC local service in our application to analyze statements and extract referenced tables and output columns.

If you don't want to do it yourself, there's this:

https://github.com/google/zetasql

Parsing is huge but it's amazing how small a part of the job it is. This library isn't even the half of it.

I mean the postgres parser (and semantic changes) for ZetaSQL. The zetasql parser is in a file called zetasql/parser/bison_parser.y, I strongly suspect they now have a file called something like zetasql/pgparser/bison_parser.y as well (and much more pervasive changes to support the deeper differences in the dialects).

This is the lexical structure and syntax docs for the new postgres inteface to cloud spanner:

https://cloud.google.com/spanner/docs/postgresql/lexical

And this is the zetasql lexical structure and syntax docs:

https://github.com/google/zetasql/blob/master/docs/lexical.m...

Notice that the new PG docs are an edit of the Zeta ones - evidence that my hypothesis is correct.

zetasql implements BigQuery, Spanner, and Dataflow dialects.