flex VS cpp-peglib

In this post, we will focus on tokenization. There are many ways to do tokenization, including using regular expressions or generating tokenizer code with generators such as flex. But what if we check how rustc's parser does it?

I am using Flex++, which is Flex for C++ and I am having trouble setting the input file. Flex++ uses the FlexLexer class provided in FlexLexer.h to create the lexer object(https://github.com/westes/flex/blob/master/src/FlexLexer.h). In my main function I have

Further applications of DFA and NFA: lex or flex, yacc or bison, and POE :-)

How about writing a programming language using Flex and Bison? There are lots of good tutorials and examples out there.

Some lexer generators (notably Flex) take input from a file handle by default. While you can always read a file into a string before passing it to the generated lexer, this is not seen as "the best" since you have to read in all the data into memory, which can be a lot.

Ages ago, I loved writing domain-specific toy languages, and almost always used flex to generate lexers and GNU bison to generate the parser. I've begun a new toy project and I don't think those two will cut it this time, so I'm looking for other tools that integrate well with C++.

I don't understand :c don't lexers like Flex work off of regex rules? Isn't this the correct first step to parse it?

I wrote a C compiler using flex [1] and bison [2]. The glue between them is a bit hacky.

At some point ANTLR [3] looked promising, but these days I'd probably write a lexer and recursive descent parser by hand, then generate LLVM IR.

[1] https://github.com/westes/flex

github.com/westes/flex/releases

Un commento in flex, uno storico software, tanto per mostrare che anche i migliori fanno le cose alla buona.

Debugging a complex PEG is a nightmarish task. I use various tools, but I couldn't find anything out there that will let you set a breakpoint in a file that's being parsed and let you explore the parsing state.

The most useful tools I found were adjacent to the cpp-peglib library: https://github.com/yhirose/cpp-peglib

This comes with a PEG playground: https://yhirose.github.io/cpp-peglib/

I really liked pegdebug: https://mqnc.github.io/pegdebug/

With sample output here: https://mqnc.github.io/pegdebug/example/output.html

pegdebug is nice for small sets of data, but it rapidly gets swamped by anything over about 50 lines.

If anyone has other suggestions for debugging PEGs, please reply and let me know,.

I used yhirose's PEG (parsing expression grammar) libraryfor parsing.

By the same author, https://github.com/yhirose/cpp-peglib is also very nice.

Most of the research into this sort of incremental and differential parsing unsurprisingly looks to be in the Haskell community, but I was hoping that someone here might know of some C++-friendly tools that might support generating parsers that support interruption and resuming. cpp-peglib looked really promising, but its parsers don't look like they can be engaged from any but the initial state.