spaCy VS CoreNLP

Hi Community, In this article, I will demonstrate below steps to create your own chatbot by using spaCy (spaCy is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython):

SpaCy: An open-source library providing tools for advanced NLP tasks like tokenization, entity recognition, and part-of-speech tagging.

spaCy https://spacy.io/

In this article, I'm going to walk through a sentiment analysis project from start to finish, using open-source Amazon product reviews. However, using the same approach, you can easily implement mass sentiment analysis on your own products. We'll explore an approach to sentiment analysis with one of the most popular Python NLP packages: spaCy.

Spacy [0] is a state-of-art / easy-to-use NLP library from the pre-LLM era. This post is the Spacy founder's thoughts on how to integrate LLMs with the kind of problems that "traditional" NLP is used for right now. It's an advertisement for Prodigy [1], their paid tool for using LLMs to assist data labeling. That said, I think I largely agree with the premise, and it's worth reading the entire post.

The steps described in "LLM pragmatism" are basically what I see my data science friends doing — it's hard to justify the cost (money and latency) in using LLMs directly for all tasks, and even if you want to you'll need a baseline model to compare against, so why not use LLMs for dataset creation or augmentation in order to train a classic supervised model?

[0] https://spacy.io/

[1] https://prodi.gy/

spaCy

I chose spacy. Although it's not state of the art, it's very well established and stable.

The Stanford CoreNLP Model

If your looking at spacy have a look at Apache OpenNLP and Core NLP.

Tokenize with NLTK, SpaCy or CoreNLP

It should not take nearly that long. However, again I must recommend you take this conversation to github

Stanford has a great program and the Stanford NLP Group maintains CoreNLP which I have used before.

So why not use the Stanford CoreNLP library?

If you need NER, there’s no need to implement it yourself. There are several popular libraries that can do this for you nowadays. Five of these libraries, Stanford CoreNLP, NLTK, OpenNLP, SpaCy, and GATE, were already mentioned in the title.

Check something like this out to start: https://stanfordnlp.github.io/CoreNLP/

You don't actually even need to learn how to do deep learning if you're doing something fairly basic, which it sounds like you are. There are a lot of good tools you can use basically straight out of the box for something like this. Check out https://huggingface.co/course/chapter1, https://course.spacy.io/en/, https://guide.allennlp.org/ and https://www.nltk.org/book/. If java's more your thing, add https://stanfordnlp.github.io/CoreNLP/ to the list.

To give a contrasting perspective, I think the Java ecosystem is much better suited for many data science tasks, and has a growing and well-maintained set of libraries for general purpose machine learning. I won't list them all, but TF-Java, DJL et al. have implementations of many modern architectures and there are a number of excellent libraries (CoreNLP, Lucene et al.) for working with text.