ANTLR VS parser

antlr https://github.com/antlr/antlr4

Download the ANTLR jar from https://www.antlr.org/download/antlr-4.9.2-complete.jar Add the ANTLR jar to your project's classpath. Install the ANTLR Kotlin target by following the instructions at https://github.com/antlr/antlr4/blob/master/doc/targets/Kotlin.md Next, you'll need a Perl grammar file for ANTLR:

Write a context-free grammar for it, commonly written in Backus Naur Form, and use that to write a parser. There are tools named "parser generators" like antlr4 that can automatically convert a BNF grammar into a parser.

Tutorial on using it in Java: https://www.baeldung.com/java-antlr Github project itself with docs and examples: https://github.com/antlr/antlr4

Depending on how complex your expressions are, you might consider using something like antlr and writing your own parser for it. Setting up something to handle math and string operations wouldn’t be very hard and then you can control the syntax however you like. You can use a visitor and visit each node in the syntax tree and return the result of each sub-expression.

> ...than it would be to learn the exact syntax and quirks and possibly bugs of someone else's implementation...

Yup. Also, having deep knowledge of the language is required.

SQLite's grammar is neat. Creating a compatible parser would make a fun project. Here's a pretty good example: https://github.com/bkiers/sqlite-parser (Actual ANTLR 4 grammar: https://github.com/bkiers/sqlite-parser/blob/master/src/main... )

Postgres, which tries to be compliant with the latest standards, however...

SQL-2016 is a beast. Not to mention all the dialects.

I'm updating my personal (soon to be FOSS) grammar from ANTLR 3 LL(k) to ANTLR 4 ALL().

I've long had a working knowledge of SQL-92, with some SQL-1999 (eg common table expressions).

But the new structures and extensions are a bit overwhelming.

Fortunately, ANTLR project has ~dozen FOSS grammars to learn from. https://github.com/antlr/grammars-v4/tree/master/sql

They mostly mechanically translate BNFs to LL(k) with some ALL(). Meaning few take advantage of left-recursion. https://github.com/antlr/antlr4/blob/master/doc/left-recursi...

Honestly, I struggled to understand these grammars. Plus, not being conversant with the SQL-2016 was a huge impediment. Just finding a succinct corbis of test cases was a huge hurdle for me.

Fortunately, the H2 Database project is a great resource. https://github.com/h2database/h2database/tree/master/h2/src/...

Now for the exciting conclusion...

My ANTLR grammar which passes all of H2's tests looks nothing like any of the official or product specific BNFs.

Further, I found discrepancy between the product specific BNFs and their implementations.

So a lot of trial & error is required for a "real world" parser. Which would explain why the professional SQL parsing tools charge money.

I still think creating a parser for SQLite is a great project.

can be used to parse

I already posted here about a project, but I could also use help on Mantle. It's a new command framework powered by ANTLR, if that's something you're interested in.

ive been tryng to work with antlr4 and go but it seems that i cant import the runtime, it says that the antlr runtime isnt in the gopath but ive already done go get github.com/antlr/antlr4/runtime/antlr4 and i dont know what to do now, im on windows if anyone knows what to do it would be very helpful. thanks already

For MySQL the situation is a bit different. sqlc uses the parser of TiDB (https://github.com/pingcap/parser), which is a parser that aims to be basically compatible with MySQL, but is quite young and is not a MySQL parser. The most basic queries work, but even simple joins or aggregations usually result in variables with unknown data types or wrong nullability. So you loose a lot of the benefits of sqlc. Manual type annotations for MySQL also do not work most of the time. They are simply ignored and forwarded to MySQL as invalid query if they do not occur on a place where sqlc is expecting them.

This is a disaster waiting to happen. Regular expressions should never be used to parse non-regular languages, of which SQL is one.

There are a variety of mature MySQL dialect parsers available[1][2], and MySQL should have its own public APIs for transforming a query into an AST. Any of those would be a safer and more correct alternative.

[1]: https://github.com/pingcap/parser

[2]: https://github.com/square/mysql-parser

Pingcap parser is a MySQL parser in Go.