locust VS gumtree

Nice!

Have been working on something similar, although my use case is more about learning how code has changed across git commits: https://github.com/bugout-dev/locust

For Javascript/Typescript/React support, like you, I hooked into the Babel toolchain. Can't recommend it highly enough.

There's also a newish project called quick-lint-js which seems to have written their own from-scratch AST parser for JS, but I haven't tried it yet: https://github.com/quick-lint/quick-lint-js

Finally, another project that I know in this space is comby (I believe it is owned/maintained by the folks at Sourcegraph): https://comby.dev/

Don't know why I dumped all those links there. Just figured there may be something useful in them for you. Am also just super passionate about building knowledge about code bases by analyzing their ASTs. Nice to meet a fellow enthusiast. :)

I maintain a free/open source project that does exactly what the author asks for: https://github.com/bugout-dev/locust.

Our tool uses git as the foundation of its functionality. It superimposes git diffs on top of ASTs.

It is insanely powerful.

For example, we use it to power semantic code search and current support Python, Javascript, and Java. We generate a JSON object defining the AST differences between initial and terminal commits on GitHub PRs and doing text search on the JSON objects performs surprisingly well when we want to answer questions like, "When did we add dateutils as a dependency?" or "When did we last change the /journals handler on the API?"

The Python integration currently sees the most use but if you are interested in other languages, we would be happy to support it.

Do drop me a DM if you want help getting started with Locust.

My team has a similar project (Locust: https://github.com/bugout-dev/locust) where the goal is to learn the semantic meanings of code changes in git commits, GitHub PRs, etc.

Since we took git diffs as a target for semantic analysis, we have a different approach to our diffs. We start with line-by-line diffs (specifically using "git diff") and then take a semantic diff by superimposing the git diff information on top of the initial and terminal ASTs.

This makes the diff calculation cheaper because we don't have to do full diff between trees.

Haven't updated the code in a few months, but my team is actively using Locust on public GitHub repos to learn the semantics of those code bases. We do plan to do some work on it soon to make it easier to make Locust easier to use (especially as a library).

Really need to sit down and take a proper look at tree-sitter. We currently support Locust diffs for Python, Javascript, and Java, but each one is custom written and implements the same basic algorithm. It looks like tree sitter might just crush this problem for us.

I'm not familiar with Pijul, and haven't finished watching this presentation, but IME the problems with modern version control tools is that they still rely on comparing lines of plain text, something we've been doing for decades. Merge conflicts are an issue because our tools are agnostic about the actual content they're tracking.

Instead, the tools should be smarter and work on the level of functions, classes, packages, sentences, paragraphs, or whatever primitive makes sense for the project and file that is being changed. In the case of code bases, they need to be aware of the language and the AST of the program. For binary files, they need to be aware of the file format and its binary structure. This would allow them to show actually meaningful diffs, and minimize the chances of conflicts, and of producing a corrupt file after an automatic merge.

There has been some research in this area, and there are a few semantic diffing tools[1,2,3], but I'm not aware of this being widely used in any VCS.

Nowadays, with all the machine learning advances, the ideal VCS should also use ML to understand the change at a deeper level, and maybe even suggest improvements. If AI can write code for me, it could surely understand what I'm trying to do, and help me so that version control is entirely hands-free, instead of having to fight with it, and be constantly aware of it, as I have to do now.

I just finished watching the presentation, and Pijul seems like an iterative improvement over Git. Nothing jumped out at me like a killer feature that would make me want to give it a try. It might be because the author focuses too much on technical details, instead of taking a step back and rethinking what a modern VCS tool should look like today.

[1]: https://semanticdiff.com/

[2]: https://github.com/trailofbits/graphtage

[3]: https://github.com/GumTreeDiff/gumtree

There’s also gumtree: https://github.com/GumTreeDiff/gumtree/wiki/Languages

GumTree does AST level diffing, hypothetically one could build VCS on top of that. That would work for binary files as long as they are parseable to some sort of sensible AST.

https://github.com/GumTreeDiff/gumtree

There are a few AST-based diffing programs e.g. GumTreeDiff. I haven't tried any of them though.