imdb-rename VS stack-graphs

I wrote a tool that did something related a while back using IMDb data: https://github.com/BurntSushi/imdb-rename

This might be of interest: https://github.com/BurntSushi/imdb-rename

What a shit take. The article itself is perhaps a nice light overview of 101-ish level concepts, although knowing how and when to apply them in a real engineering context is not something I would consider 101 level. And certainly, building something that is actually at the scale of GitHub Search is nowhere near 101 level.

This is what a 101-level inverted index implementation looks like: https://github.com/BurntSushi/imdb-rename

In other words, absolutely nothing like what GitHub built. Nowhere close.

imdb-rename is an example of a tool that memory maps FSTs on disk in order to execute fulltext searches very quickly on the command line.

> How else would you lazy-load a database of (say) 32GB into memory, almost instantly?

That's what the fst crate[1] does. It's likely working at a lower level of abstraction than you intend. But the point is that it works, is portable and doesn't require any cooperation from the OS other than the ability to memory map files. My imdb-rename tool[2] uses this technique to build an on-disk database for instantaneous searching. And then there is the regex-automata crate[3] that permits deserializing a regex instantaneously from any kind of slice of bytes.[4]

I think you should maybe provide some examples of what you're suggesting to make it more concrete.

[1] - https://crates.io/crates/fst

[2] - https://github.com/BurntSushi/imdb-rename

[3] - https://crates.io/crates/regex-automata

[4] - https://docs.rs/regex-automata/0.1.9/regex_automata/#example...

https://github.com/pyjarrett/septum

The hardest part about getting code search right imo is grabbing the right amount of surrounding context, which septum is aimed at solving on a per-file basis.

Another one I'm surprised hasn't been mentioned is stack-graphs (https://github.com/github/stack-graphs), which tries to incrementally resolve symbolic relationships across the whole codebase. It powers github's cross-file precise indexing and conceptually makes a lot of sense, though I've struggled to get the open source version to work

[2]: https://github.com/github/stack-graphs

> It doesn't have the faintest idea where the name is defined, or if there's even a difference between a function name, a parameter name, or a word in a comment.

I don't think what you are saying is actually true for stack-graphs[0][1].

[0]: https://github.com/github/stack-graphs

[1]: https://github.blog/2021-12-09-introducing-stack-graphs/

I think GitHub still has some use for tree-sitter. In this post it's mentioned that their new code navigation system is based on tree-sitter. In a more recent post they welcome contributers to add special code navigation queries to existing languages. You can find their public repository here if you want to follow along with any developments. Since their code navigation system relies heavily on tree-sitter I don't think it's going anywhere soon (fingers crossed).

Which they implement in Rust. https://github.com/github/stack-graphs

As mentioned elsewhere on this thread, stack graphs and Semantic were built by the same team (which I manage). Semantic is not abandoned, we've just been focusing on a different layer of our tech stack for the past year or so. https://news.ycombinator.com/item?id=29501389

That PR on the Semantic repo was our first attempt at implementing these ideas. We decided to reimplement it in a separate library (also open source, https://github.com/github/stack-graphs), which only builds on tree-sitter directly so that there's an easier story for us and language communities to add support for new languages. It's a fair point that we could have closed the Semantic PR to indicate that more clearly.