nearley VS IParse

While I suspect I would learn more writing a tokenizer and parsing logic myself I find grammars much easier to read and maintain.

ANTLR is pretty good and is supported across several languages and something I had previously used for some quick Elasticsearch query syntax munging in Python. It also means you can often start from an already existing grammar.

The JS version of ANTLR didn't seem to work for me so for the SQL/JSONPath stuff ended up using the Moo lever and Nearly parser which was rather pleasant. https://nearley.js.org

Parsers in nearley.js [1] are written in a very readable EBNF-like DSL; then they get desugared down to a JS file that's a lot like your snippet.

[1] https://github.com/kach/nearley

Nice library with many features. But I do not always understand the focus on memory usage. I guess that the reason behind this is that less memory allocations, have a positive effect on execution times. In a parser, where you often have to compare identifiers, it is a good idea to put all strings for identifiers into a unique pointer with the help of a hash table.

In my interpreting parser [1] I use a hexa hash tree [2] for storing identifiers. It is not very memory efficient, but very fast. It turns every string (from the input buffer) into a unique pointer for that string pointing to a copy of the string. In this way comparing string (identifiers) is equivalent to comparing pointers.

The idea of the hexa hash tree is that is a tree where each node has sixteen child nodes. Which node is selected is based on a step wise evaluated hash function that first takes the lower four bytes of the string, and after reaching the end of the string, the higher four bytes of the string. The nodes often taken up more memory space than the strings themselves.

[1] https://github.com/FransFaase/IParse/

[2] https://github.com/FransFaase/IParse/blob/master/software/Id...

I wrote a tool that can process a number of MarkDown files with fragments of C code and put all those fragments in the right order to produce a file that can be compiled. It is grammar based and works with manipulating Abstract Syntax Trees, so I guess, it could be adapted for different programming languages. See: https://github.com/FransFaase/IParse#markdownc

Interesting. I have not looked into the code, but I wonder whether it is a compiler, or just an interpreter, e.g. it converts the grammar into some internal representation that is executed by an interpreter or virtual machine. I started worked on an interpreting parser in C many years ago. And later also made Java, C++ and JavaScript version of it. For the JavaScript implementation, see: https://fransfaase.github.io/ParserWorkshop/Online_inter_par... For the C++ version, see: https://github.com/FransFaase/IParse

I implemented an unparse function in IParse, which is not a parser generator, but a parser that interprets a grammar. See for example https://github.com/FransFaase/IParse/blob/master/software/c_... where symbols starting with a back slash are a kind of white space terminals during the unparse. For example, \inc stands for incrementing the indentation where \dec decrements it. The \s is used to indicate that at given location a space should be included.