LibCST VS RedBaron

LibCST: Seems to only show in terminal.

While you can do this using ANTLR or any other lexing/parsing tool, it's honestly a bit of a pain. Whitespace and comments can go almost anywhere, even in the middle of expressions, so the grammar ends up becoming fairly messy. So, I'd recommend using a library that handles this for you, if at all possible. For example, if I wanted to code-mod Python I'd prob just use the LibCST library.

I recommend https://github.com/Instagram/LibCST (which is currently implementing rust bindings)

I've been roughly copying some of the logic from the LibCST project. This struct in particular- https://github.com/Instagram/LibCST/blob/main/native/libcst/src/tokenizer/text_position/char_width.rs does a good job of normalizing the line endings of a str. The long way around you could mimic this construct, transform the str to normalized line endings, and then split on "\n" or make a somewhat more complicated transformer which turns a large str into a Vec (or Vec).

I myself am working on upgrading LibCST's parser engine to support the new syntax, and then am hoping we can rewrite Black's formatting rules in terms of LibCST's API. That's not a small amount of work, which is why we can't confidently say that's going to be the way forward.

Never used it but it appears that Facebook/Instagram have a format preserving CST library for Python: https://github.com/Instagram/LibCST

https://github.com/PyCQA/redbaron

E.g. PyCQA/bandit does static analysis for security issues in Python code: