semantic-source VS Glean

One production example I know: GitHub code navigation is written in Haskell https://github.com/github/semantic

ah, easy. it's because support has not been added into https://github.com/github/semantic which is the tech that powers the GitHub UI. Adding support is pretty easy/mainly glue code [1] that imports the tree sitter API.

[1] https://github.com/github/semantic/blob/793a876ae45d38a6bd17...

I believe they use semantic (Haskell program that uses tree-sitter) for navigation: https://github.com/github/semantic

So the answer may be that semantic does not yet have support for the language in question.

GitHub used Haskell for implementing Semantic, a command-line tool for parsing, analyzing, and comparing source code.

As far as engineering effort, you can read this GitHub comment for an overview of where we’d like to take the project in the future. The tl;dr here is that the open sum type view of the world made it very concise to fold over syntax trees (since such a view of data is ultimately unityped, recursion schemes Just Work), but the tradeoff thus associated—namely, that you have to parse a concrete syntax tree into an open-sum view (a complicated and painful-to-read process), that you can never really be sure how a given syntax tree is shaped, and that the types don’t help you nearly as much as they could—proved to be too onerous to deal with. Going forward, we’re generating syntax types from the AST once per target language, and working on an abstraction (probably via this generated code; I made five separate efforts at using Generics for this, and failed every time) that recovers at least some of the convenience of recursion schemes. It turns out that recursion schemes over a mutually recursive syntax tree—as pretty much every language’s syntax trees are, in practice—are pretty much an unsolved problem, especially when extended to languages like TypeScript, which have hundreds of different syntax nodes.

I'm just curious. It seems there hasn't been much activity in https://github.com/github/semantic Is GitHub still using semantic it to power some code navigation features? Has it been abandoned or is there some successor project that has taken its place? Is there any writeup / lessons learned about this project?

is this from Github semantic (https://github.com/github/semantic)?

Seems very suspicious since it’s the same goal using the same technologies. The latest commit is 4mo ago but i assume they have a closed-source version they’ve been working on.

Meanwhile their Tree-Sitter-based semantic parser[1] looks abandoned. There is even rotting for years pull request[2] adding support of the same stack graphs into it.

[1] https://github.com/github/semantic

[2] https://github.com/github/semantic/pull/535

Even if it is true (which is an unsubstantiated gossip at the moment), Sigma is not the only Haskell project at Facebook. See https://github.com/facebookincubator/Glean for another example.

Meta uses something very much like this in production. It’s open sourced at https://glean.software/

Glean, a system for collecting and working with facts about source code.

Glean was recently open sourced, it might fit your needs - https://glean.software/

Cool! I would love to play around with this.

How do I write a schema and indexer for my favorite programming language that isn't currently (and won't be) supported with official releases?

For Schemas, [1] says to modify (or base new ones off) these: https://github.com/facebookincubator/Glean/tree/main/glean/s...

For Indexers, it's a little less clear but it looks like I need to write my own type checker?

[1] https://glean.software/docs/schema/workflow

Kythe has one schema, whereas with Glean each language has its own schema with arbitrary amounts of language-specific detail. You can get a language-agnostic view by defining an abstraction layer as a schema. Our current (work in progress) language-agnostic layer is called "codemarkup" https://github.com/facebookincubator/Glean/blob/main/glean/s...

For wiring up the indexer, there are various methods, it tends to depend very much on the language and the build system. For Flow for example, Glean output is just built into the typechecker, you just run it with some flags to spit out the Glean data. For C++, you need to get the compiler flags from the build system to pass to the Clang frontend. For Java the indexer is a compiler plugin; for Python it's built on libCST. Some indexers send their data directly to a Glean server, others generate files of JSON that get sent using a separate command-line tool.

References use different methods depending on the language. For Flow for example there is a fact for an import that matches up with a fact for the export in the other file. For C++ there are facts that connect declarations with definitions, and references with declarations.

There will be more indexers: we have Python, C++/Objective C, Rust, Java and Haskell. It's just a case of getting them ready to open source. You can see the schemas for most of these already in the repo: https://github.com/facebookincubator/Glean/tree/main/glean/s...

Existing schemas are here: https://github.com/facebookincubator/Glean/tree/main/glean/schema/source