Apache Solr VS Toshi

There are already many project about search:

- https://www.marginalia.nu/

- https://searchmysite.net/

- https://lucene.apache.org/

- elastic search

- https://presearch.com/

- https://stract.com/

- https://wiby.me/

I think that all project are fun. I would like to see one succeeding at reaching mainstream level of attention.

I have also been gathering links meta data for some time. Maybe I will use them to feed any eventual self hosted search engine, or language model, if I decide to experiment with that.

- domains for seed https://github.com/rumca-js/Internet-Places-Database

- bookmarks seed https://github.com/rumca-js/RSS-Link-Database

- links for year https://github.com/rumca-js/RSS-Link-Database-2024

Elasticsearch is based on Lucene and is used by various companies and developers across the world to build custom search solutions.

elastic search is kinda heavyweight infra for a small project. Its built on top of apache lucene (https://lucene.apache.org), which you can use directly.

Elasticsearch is based on Lucene, which is built in Java. This means that monitoring the Java Virtual Machine (JVM) memory is crucial to understand the current usage of the whole system.

Apache Lucene which seems to have a lot more features than Elasticsearch

Generally with term vectors and a tf-idf index. Lucene is a good starting place to help.

try elasticsearch or solr, behind the scenes they both use https://lucene.apache.org/ if you don't want basically a full nosql database service, but I'd just slap solr up and call it a day.

OpenSearch is an open-source database to ingest, search, visualize, and analyze data. It’s built on top of Apache Lucerce, a FOSS library for indexing and search, which OpenSearch leverages for more advanced analytics capabilities, like anomaly detection, machine learning, full-text search, and more.

Lucene is the thing you think you need. Elastic Search is a nice wrapper for it. But these are Java, so maybe you want Sphinx Search (C++) or MeiliSearch (Rust).

I don't think you have an active project that addresses all those use cases. There was an attempt in Rust with Toshi that is built on top of tantivy, but the project seems to have stalled.

I wish we had an extension like ZomboDB but using a lighter search engine like https://github.com/quickwit-oss/quickwit, https://github.com/toshi-search/Toshi and https://github.com/mosuka/bayard

Here I'm listing engines based on https://github.com/quickwit-oss/tantivy - tantivy is comparable to Lucene in its scope - but I'm sure there are other engines that could tackle ElasticSearch.

Another thing that could happen is maybe directly embed tantivy in Postgres using an extension, perhaps this could be an option too.

I do something similar, though I index the page myself via a little browser extension I wrote. I click a button, the content gets POSTed to a server that throws it in Toshi[1]. I hacked it together on a Saturday, and it's been pretty handy; as you describe, much more useful than any bookmarking approach I've tried before.

[1] https://github.com/toshi-search/Toshi

There is some incredible work being done in the web department, frameworks like rocket.rs and actix.rs are amazing. To get the latest info on web development in Rust, check arewewebyet.org. It doesn't list Toshi though, which is weird.