NLTK VS spaCy

Using the NTLK Natural Language Toolkit

Is this just using LLM to be cool? How does pure LLM with simple "In the scale between 0-10"" stack up against traditional, battle-tested sentiment analysis tools?

Gemini suggests NLTK and spaCy

https://www.nltk.org/

https://spacy.io/

alternatively, could we not simply split by common characters such as newlines and periods, to split it within sentences? it would be fragile with special handling required for numbers with decimal points and probably various other edge cases, though.

there are also Python libraries meant for natural language parsing[0] that could do that task for us. I even see examples on stack overflow[1] that simply split text into sentences.

[0]: https://www.nltk.org/

Check out https://www.nltk.org/ and work through it, it'll give you a foundational understanding of how all this works, but very basically it's just a fancy auto-complete.

NLTK Documentation

If this is True, then you need also Natural Language Toolkit to process the words.

In that case, you should definitely have a look at Python's nltk library which stands for Natural Language Toolkit. They have a rich corpus collection for all kinds of specialized things like grammars, taggers, chunkers, etc.

Is this just using LLM to be cool? How does pure LLM with simple "In the scale between 0-10"" stack up against traditional, battle-tested sentiment analysis tools?

Gemini suggests NLTK and spaCy

https://www.nltk.org/

https://spacy.io/

Hi Community, In this article, I will demonstrate below steps to create your own chatbot by using spaCy (spaCy is an open-source software library for advanced natural language processing, written in the programming languages Python and Cython):

SpaCy: An open-source library providing tools for advanced NLP tasks like tokenization, entity recognition, and part-of-speech tagging.

spaCy https://spacy.io/

In this article, I'm going to walk through a sentiment analysis project from start to finish, using open-source Amazon product reviews. However, using the same approach, you can easily implement mass sentiment analysis on your own products. We'll explore an approach to sentiment analysis with one of the most popular Python NLP packages: spaCy.

Spacy [0] is a state-of-art / easy-to-use NLP library from the pre-LLM era. This post is the Spacy founder's thoughts on how to integrate LLMs with the kind of problems that "traditional" NLP is used for right now. It's an advertisement for Prodigy [1], their paid tool for using LLMs to assist data labeling. That said, I think I largely agree with the premise, and it's worth reading the entire post.

The steps described in "LLM pragmatism" are basically what I see my data science friends doing — it's hard to justify the cost (money and latency) in using LLMs directly for all tasks, and even if you want to you'll need a baseline model to compare against, so why not use LLMs for dataset creation or augmentation in order to train a classic supervised model?

[0] https://spacy.io/

[1] https://prodi.gy/

spaCy