Gigablast VS zr

Gigablast - A search engine whose source code is available on Github. TMK, it doesn't depend on any existing indexes either and uses its own crawler. Would be what I'd recommend to everyone but they recently collaborated with a company that has a questionable past to create a privacy focused search engine. A company that was behind the hostile takeover of Freenode IRC.

Any new open-source search option is good, but I also wish more attention was given to prior open projects like GigaBlast[0]/KBlast[1] crawlers, etc.

It hasn't escaped the wider world that quality open-source search is desirable, and it's hard to think what this new EU project brings to the table that isn't already available if others want to contribute to existing efforts. I wish the EU project the best of luck of course!

[0] https://github.com/gigablast/open-source-search-engine

[1] https://github.com/fossabot/kblast

Gigablast Source Apache-2.0

You could look at the source code for Gigablast. https://github.com/gigablast/open-source-search-engine

Amazing!

I have only one recommendation that might make the search a bit more relevant, e.g when searching for 'linux locking' or 'kernel locking' kind of things.

Try to upsort things that match near the top of the content, like the top of the man page vs middle vs bottom.

One easy way to do it without having to store the positions, is to index the ngrams with max(sqrt,8) of their line number, this will cover first 64 lines, you can also use log() or just decide ad hock, top, middle, bottom of the document, so you can use only 3 values.

e.g. https://www.kernel.org/doc/html/v5.0/kernel-hacking/locking.... would do unreliable_1 guide_1 locking_1 ... then at line 4 kernel_2 locking_2 ... after line 50 ... then_7 ... and after that everything will be _8.

then just make the query "kernel locking" to "dismax(kernel_1 OR kernel_2 OR kernel_3...) AND dismax(locking_1 OR locking_2 ...) with some tiebreaker of 0.1 or so, you can also say "i want to upsort things on the same line, or few lines apart" by modifying the query a bit.

It works really well and costs very little in terms of space, i tried it at https://github.com/jackdoe/zr while searching all of stackoverfow/man pages and etc and was pretty surprised by the result.

This approach is a bit cheaper than storing the positions because positions are (lets say) 4 bytes per term per doc, while this approach has fixed uppre bound cost of 8*4 per document (assuming 4 byte document ids)