pg-query-clj VS libpg_query

> One of the things that puts metaprogramming on wheels is being able to pull in grammars, schemas and such for external systems.

That's what I've been trying to do with: https://github.com/schemamap/schemamap

For a well-constrained Postgres schema, a deterministic SQL compiler can be built (with plenty of logic programming/constraint-solving/hypergraph-analysis) that can integrate arbitrary external systems data.

While this is NP-hard, with some clever use of embeddings and ontologies, and use of every single constraint AST within the DB to reduce the search space, this becomes feasible problem.

For any Clojurists interested, I've packaged `pg_query`, so you can use it in your applications: https://github.com/schemamap/pg-query-clj

At the moment I'm saving up for the next development cycle, so not only PG->PG schema mappings can be solved for (JSON schema is next!). Hope this sounds interesting :)

pgfence doesn't use regex to guess at SQL patterns. It uses libpg_query — PostgreSQL's actual parser, compiled to a C library and exposed via Node.js bindings. The same parser that Postgres itself uses to understand your SQL.

For tooling developers, the libpg_query project extracts PostgreSQL's actual parser code, providing 100% compatibility with PostgreSQL's SQL parsing behavior. For MySQL, the most widely used parser is from TiDB, but it has compatibility differences since it was built specifically for TiDB's needs rather than pure MySQL compatibility.

Yes, the same way. It's all based on the extremely useful `https://github.com/pganalyze/libpg_query` project, which is where the "extracted the parser from Postgres" part comes in.

Supabase's LSP also uses tree-sitter for corrections and autocomplete, because one drawback of using the server's source is that pg_query only works on well-formed/executable SQL - when it detects a malformed query, it formulates an error and exits, since that's what you want in an SQL server. So for partially-correct syntax and fill-forward, tree-sitter covers the gaps.

This is why we decided not to create a custom parser. Instead, we leverage the existing libpg_query library to parse SQL code reliably. The pganalyze team has published a great blog post on why this approach is preferred.

libpg_query is a good library!

"C library for accessing the PostgreSQL parser outside of the server environment"

https://github.com/pganalyze/libpg_query?tab=readme-ov-file#...

  pg_query wrappers in other languages:

Finally, we decided to use a different library - libpg_query. This is a C library for parsing the query using the Postgres code. The library has a JavaScript binding in the form of the libpg-query library which was very convenient to use.

This in combination with [pg_query](https://github.com/pganalyze/libpg_query) could be a very powerful combination that allows writing generic static analyzers.

It's true that the core PG code isn't written in a modular way that's friendly to integration piecemeal in other projects (outside of libpq).

For THIS PARTICULAR case, the pganalyze team has actually extracted out the parser of PG for including in your own projects:

https://github.com/pganalyze/libpg_query

Generally I agree that this would be great to have, and Postgres does have a set of libraries it already maintains as part of the main source tree (i.e. libpq, etc), and there is a shared set of code between the backend and the "frontend" (https://github.com/postgres/postgres/tree/master/src/common). So theoretically you could imagine the parser moving into that shared code portion, sharing code but not necessarily requiring linking to a library from the backend.

However, the challenge from what I've understood from past conversations with some folks working on Postgres core is that the parser is currently heavily tied into the backend - note the parser isn't just the scan.l/gram.y file, but also the raw parse node structs that it outputs. You can see how many files we pull in from the main tree that are prefixed with "src_backend": https://github.com/pganalyze/libpg_query/tree/15-latest/src/...

Further, there isn't a canonical way to output node trees into a text format today in core, besides the rather hard to work with output of debug_print_parse - there have been discussions on -hackers to potentially utilize JSON here, which may make this a bit easier. Note that in libpg_query we currently use Protobuf (but used to use JSON), which does have the benefit of getting auto-generated structs in the language bindings - but Protobuf is not used in core Postgres at all today.

All in all, I think there is some upstream interest, but its not clear that this is a good idea from a maintainability perspective.