NLTK VS Jieba

Using the NTLK Natural Language Toolkit

Is this just using LLM to be cool? How does pure LLM with simple "In the scale between 0-10"" stack up against traditional, battle-tested sentiment analysis tools?

Gemini suggests NLTK and spaCy

https://www.nltk.org/

https://spacy.io/

alternatively, could we not simply split by common characters such as newlines and periods, to split it within sentences? it would be fragile with special handling required for numbers with decimal points and probably various other edge cases, though.

there are also Python libraries meant for natural language parsing[0] that could do that task for us. I even see examples on stack overflow[1] that simply split text into sentences.

[0]: https://www.nltk.org/

Check out https://www.nltk.org/ and work through it, it'll give you a foundational understanding of how all this works, but very basically it's just a fancy auto-complete.

NLTK Documentation

If this is True, then you need also Natural Language Toolkit to process the words.

In that case, you should definitely have a look at Python's nltk library which stands for Natural Language Toolkit. They have a rich corpus collection for all kinds of specialized things like grammars, taggers, chunkers, etc.

Let's continue with jieba as an example. This is the main program logic for pg_jieba, which is also a Python package, so let's use Python for the example.

jieba to do Chinese word segmentation

Jieba: Chinese text segmenter

If you know some python you can use a popular library called Jieba 结巴 to automatically get pinyin for every word. (Jieba has actually been ported to many languages) You can also use it to break a chinese text into a set of unique words for easy addition to your spreadsheet.

Since you're saying the main issue is segmentation, there are libraries to help out with that issue. jieba is fantastic if you have a Python backend, nodejieba (50k downloads/week) if it's more JS-side.

https://github.com/fxsjy/jieba/ (has some good word frequency data)