semantic-source VS stack-graphs

One production example I know: GitHub code navigation is written in Haskell https://github.com/github/semantic

ah, easy. it's because support has not been added into https://github.com/github/semantic which is the tech that powers the GitHub UI. Adding support is pretty easy/mainly glue code [1] that imports the tree sitter API.

[1] https://github.com/github/semantic/blob/793a876ae45d38a6bd17...

GitHub used Haskell for implementing Semantic, a command-line tool for parsing, analyzing, and comparing source code.

As far as engineering effort, you can read this GitHub comment for an overview of where we’d like to take the project in the future. The tl;dr here is that the open sum type view of the world made it very concise to fold over syntax trees (since such a view of data is ultimately unityped, recursion schemes Just Work), but the tradeoff thus associated—namely, that you have to parse a concrete syntax tree into an open-sum view (a complicated and painful-to-read process), that you can never really be sure how a given syntax tree is shaped, and that the types don’t help you nearly as much as they could—proved to be too onerous to deal with. Going forward, we’re generating syntax types from the AST once per target language, and working on an abstraction (probably via this generated code; I made five separate efforts at using Generics for this, and failed every time) that recovers at least some of the convenience of recursion schemes. It turns out that recursion schemes over a mutually recursive syntax tree—as pretty much every language’s syntax trees are, in practice—are pretty much an unsolved problem, especially when extended to languages like TypeScript, which have hundreds of different syntax nodes.

Meanwhile their Tree-Sitter-based semantic parser[1] looks abandoned. There is even rotting for years pull request[2] adding support of the same stack graphs into it.

[1] https://github.com/github/semantic

[2] https://github.com/github/semantic/pull/535

The semantic team at GitHub uses it for statically analyzing the dozens of languages that end up in GitHub repositories: https://github.com/github/semantic/blob/eaf13783838861fe5eb6cd46d59354774a8eb88d/docs/why-haskell.md

Yesterday I was looking for some examples of projects using tree-sitter (which is C) when I found GitHub's semantic, used to analyze and compare source code, and written in Haskell: https://github.com/github/semantic/

https://github.com/pyjarrett/septum

The hardest part about getting code search right imo is grabbing the right amount of surrounding context, which septum is aimed at solving on a per-file basis.

Another one I'm surprised hasn't been mentioned is stack-graphs (https://github.com/github/stack-graphs), which tries to incrementally resolve symbolic relationships across the whole codebase. It powers github's cross-file precise indexing and conceptually makes a lot of sense, though I've struggled to get the open source version to work

[2]: https://github.com/github/stack-graphs

> It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comment.

I don't think what you are saying is actually true for stack-graphs[0][1].

[0]: https://github.com/github/stack-graphs

[1]: https://github.blog/2021-12-09-introducing-stack-graphs/

I think GitHub still has some use for tree-sitter. In this post it's mentioned that their new code navigation system is based on tree-sitter. In a more recent post they welcome contributers to add special code navigation queries to existing languages. You can find their public repository here if you want to follow along with any developments. Since their code navigation system relies heavily on tree-sitter I don't think it's going anywhere soon (fingers crossed).

Which they implement in Rust. https://github.com/github/stack-graphs

As mentioned elsewhere on this thread, stack graphs and Semantic were built by the same team (which I manage). Semantic is not abandoned, we've just been focusing on a different layer of our tech stack for the past year or so. https://news.ycombinator.com/item?id=29501389

That PR on the Semantic repo was our first attempt at implementing these ideas. We decided to reimplement it in a separate library (also open source, https://github.com/github/stack-graphs), which only builds on tree-sitter directly so that there's an easier story for us and language communities to add support for new languages. It's a fair point that we could have closed the Semantic PR to indicate that more clearly.