semantic-source VS Glean

One production example I know: GitHub code navigation is written in Haskell https://github.com/github/semantic

ah, easy. it's because support has not been added into https://github.com/github/semantic which is the tech that powers the GitHub UI. Adding support is pretty easy/mainly glue code [1] that imports the tree sitter API.

[1] https://github.com/github/semantic/blob/793a876ae45d38a6bd17...

GitHub used Haskell for implementing Semantic, a command-line tool for parsing, analyzing, and comparing source code.

As far as engineering effort, you can read this GitHub comment for an overview of where we’d like to take the project in the future. The tl;dr here is that the open sum type view of the world made it very concise to fold over syntax trees (since such a view of data is ultimately unityped, recursion schemes Just Work), but the tradeoff thus associated—namely, that you have to parse a concrete syntax tree into an open-sum view (a complicated and painful-to-read process), that you can never really be sure how a given syntax tree is shaped, and that the types don’t help you nearly as much as they could—proved to be too onerous to deal with. Going forward, we’re generating syntax types from the AST once per target language, and working on an abstraction (probably via this generated code; I made five separate efforts at using Generics for this, and failed every time) that recovers at least some of the convenience of recursion schemes. It turns out that recursion schemes over a mutually recursive syntax tree—as pretty much every language’s syntax trees are, in practice—are pretty much an unsolved problem, especially when extended to languages like TypeScript, which have hundreds of different syntax nodes.

Meanwhile their Tree-Sitter-based semantic parser[1] looks abandoned. There is even rotting for years pull request[2] adding support of the same stack graphs into it.

[1] https://github.com/github/semantic

[2] https://github.com/github/semantic/pull/535

The semantic team at GitHub uses it for statically analyzing the dozens of languages that end up in GitHub repositories: https://github.com/github/semantic/blob/eaf13783838861fe5eb6cd46d59354774a8eb88d/docs/why-haskell.md

Yesterday I was looking for some examples of projects using tree-sitter (which is C) when I found GitHub's semantic, used to analyze and compare source code, and written in Haskell: https://github.com/github/semantic/

I've responded on reddit before (https://www.reddit.com/r/rust/comments/1eqqwa7/comment/lhwwn...), but I'll restate and cover some other things here.

> I have a maybe wrong and bad opinion that LSP is actually at the wrong level. Right now every language needs to implement a from scratch implementation of their LSP server. These implementations are HUGE and take YEARS to develop. rust-analyzer is over 365,000 lines of code. And every language has their own massive, independent implementation.

rust-analyzer a big codebase, but it's also less problematic than the raw numbers would make you think. rust-analyzer has a bunch of advanced functionality (term search https://github.com/rust-lang/rust-analyzer/pull/16092 and refactors), assists (nearly 20% of rust-analyzer!) and tests.

> I think there should be a common Intellisense Database file format for providing LSP or LSP-like capabilities. Ok sure there will still be per-language work to be done to implement the IDB format.

I think you might be describing formats like (https://code.visualstudio.com/blogs/2019/02/19/lsif) and SCIP (https://github.com/sourcegraph/scip). I personally like SCIP a bit more LSIF because SCIP's design makes it substantially easier to incrementally update a large index. We use SCIP with Glean (https://glean.software/) at work; it's pretty nice.

> But you'd get like 95% of the implementation for free for any LLVM language. And generating a common IDB format should be a lot simpler than implementing a kajillion LSP protocols.

I wouldn't say 95%. SCIP/LSIF can do the job for navigation, but that's only a subset of what you want from an IDE. For example:

Even if it is true (which is an unsubstantiated gossip at the moment), Sigma is not the only Haskell project at Facebook. See https://github.com/facebookincubator/Glean for another example.

Meta uses something very much like this in production. It’s open sourced at https://glean.software/

Glean, a system for collecting and working with facts about source code.

Glean was recently open sourced, it might fit your needs - https://glean.software/