python-dbg VS rust-sitter

I've done something similar in Python: A Python debugger from scratch in Python

- https://github.com/parttimenerd/python-dbg/

This is a great series!

I noticed that the author was using https://github.com/hydro-project/rust-sitter as a parser. Which is based on https://tree-sitter.github.io/tree-sitter/. I've been hearing about Tree-sitter a lot recently, so I dug into it.

Tree-sitter is a tool for generating fast, incremental parsers. In particular, the algorithm is suited towards writing "language servers" for IDEs, which re-parse code incrementally as the user works. These kinds of incremental parsers have historically been a huge problem. It looks like Tree-sitter is an enormous practical advance in this area.

And discovering that there's a way to use Tree-sitter from Rust is fantastic. From the post:

    #[rust_sitter::language]

Oh I totally agree with your sentiment about tree-sitter. That's why I want it to be extended in functionality. It makes so much sense to have a single place where one parser can be written and everybody benefits. Much like language servers.

Where I disagree is that IMO, tree-sitter already is very close to this ideal model. It has incremental parsing. It has great tree querying. Where it needs help is an AST facade over the raw syntax tree, which is very much feasible. rust-sitter[1] does it for instance. Tree-editing and tree construction is also very much doable. I don't think it'd have an impact on grammar construction at all. As for error recovery, I think it could function as a reparsing feature where you can drop down to a manual parser (or even a secondary grammar) that is more tolerant. Or an error recovery function that can be written in any language. tree-sitter already has the ability to use a manual lexer written in native code, so this is not such a stretch.

[1]: https://github.com/hydro-project/rust-sitter.