Apache Solr VS CoreNLP

There are already many project about search:

- https://www.marginalia.nu/

- https://searchmysite.net/

- https://lucene.apache.org/

- elastic search

- https://presearch.com/

- https://stract.com/

- https://wiby.me/

I think that all project are fun. I would like to see one succeeding at reaching mainstream level of attention.

I have also been gathering links meta data for some time. Maybe I will use them to feed any eventual self hosted search engine, or language model, if I decide to experiment with that.

- domains for seed https://github.com/rumca-js/Internet-Places-Database

- bookmarks seed https://github.com/rumca-js/RSS-Link-Database

- links for year https://github.com/rumca-js/RSS-Link-Database-2024

Elasticsearch is based on Lucene and is used by various companies and developers across the world to build custom search solutions.

elastic search is kinda heavyweight infra for a small project. Its built on top of apache lucene (https://lucene.apache.org), which you can use directly.

Elasticsearch is based on Lucene, which is built in Java. This means that monitoring the Java Virtual Machine (JVM) memory is crucial to understand the current usage of the whole system.

Apache Lucene which seems to have a lot more features than Elasticsearch

Generally with term vectors and a tf-idf index. Lucene is a good starting place to help.

try elasticsearch or solr, behind the scenes they both use https://lucene.apache.org/ if you don't want basically a full nosql database service, but I'd just slap solr up and call it a day.

OpenSearch is an open-source database to ingest, search, visualize, and analyze data. It’s built on top of Apache Lucerce, a FOSS library for indexing and search, which OpenSearch leverages for more advanced analytics capabilities, like anomaly detection, machine learning, full-text search, and more.

Lucene is the thing you think you need. Elastic Search is a nice wrapper for it. But these are Java, so maybe you want Sphinx Search (C++) or MeiliSearch (Rust).

The Stanford CoreNLP Model

If your looking at spacy have a look at Apache OpenNLP and Core NLP.

Tokenize with NLTK, SpaCy or CoreNLP

It should not take nearly that long. However, again I must recommend you take this conversation to github

Stanford has a great program and the Stanford NLP Group maintains CoreNLP which I have used before.

So why not use the Stanford CoreNLP library?

If you need NER, there’s no need to implement it yourself. There are several popular libraries that can do this for you nowadays. Five of these libraries, Stanford CoreNLP, NLTK, OpenNLP, SpaCy, and GATE, were already mentioned in the title.

Check something like this out to start: https://stanfordnlp.github.io/CoreNLP/

You don't actually even need to learn how to do deep learning if you're doing something fairly basic, which it sounds like you are. There are a lot of good tools you can use basically straight out of the box for something like this. Check out https://huggingface.co/course/chapter1, https://course.spacy.io/en/, https://guide.allennlp.org/ and https://www.nltk.org/book/. If java's more your thing, add https://stanfordnlp.github.io/CoreNLP/ to the list.

To give a contrasting perspective, I think the Java ecosystem is much better suited for many data science tasks, and has a growing and well-maintained set of libraries for general purpose machine learning. I won't list them all, but TF-Java, DJL et al. have implementations of many modern architectures and there are a number of excellent libraries (CoreNLP, Lucene et al.) for working with text.