resin VS pisa

I've built a search engine. It has its own query language, data structures and binary file formats, it's MIT licensed and it has around 60 forks. Nobody uses it though, even though it has been around for some years but it was just until recently that I managed to solve the very last of the most crucial of bugs, so I don't find that surprising at all.

It works well now, though, for a Wikipedia sized text based corpus, even though it's still in beta and contains code that can still be optimized. However, before I want to go any further with the project, I'd like to see if I can sell it, the copyright to my code, that is. Because maybe I want to be in the business of creating smart code, then sell it, then move on to the next thing? And maybe some company would like to have a search engine in their software portfolio? Suppose we meet, have a drink, see what happens.

Do I have the rights to sell it, though?

I'm the author of 99.999% of the commits.

https://github.com/kreeben/resin

The EF core algorithm implemented in folly [3] may be a bit faster, and implementing partitioning on top of that is relatively easy.

It would definitely compress much better than roaring bitmaps. In terms of performance, it depends on the access patterns. If very sparse (large jumps) PEF would likely be faster, if dense (visit a large fraction of the bitmap) it'd be slower.

It is possible to squeeze a bit more compression out of PEF by introducing a chunk type for Elias-Fano of the chunk complement (for very dense chunks), but you lose the operation of skipping to a given position, which is however not needed in inverted indexes (you only need to skip past a given id, and that can be supported efficiently). That is not mentioned in the paper because at the time I thought the skip-to-position operation was a non-negotiable.

[1] https://github.com/ot/ds2i/

[2] https://github.com/pisa-engine/pisa

[3] https://github.com/facebook/folly/blob/main/folly/experiment...