flashtext VS gensim

I have some other comment on this thread where I point out why I don’t think it’s superficial. Would love to get your feedback on that if you feel like spending more time on this thread.

But it’s not obscure? FlashText was a somewhat popular paper at the time (2017) with a popular repo (https://github.com/vi3k6i5/flashtext). Their paper was pretty derivative of Aho-Corasick, which they cited. If you think they genuinely fucked up, leave an issue on their repo (I’m, maybe to your surprise lol, not the author).

Anyway, I’m not a fan of the whatabboutery here. I don’t think OG’s paper is up to snuff on its lit review - do you?

The library flashtext basically creates these tries based on keywords you give it.

Seems like https://github.com/vi3k6i5/flashtext would be better suited here.

I started developing this tool after using haystack. Pipelines are easier to build with cherche because of the operators. Also, cherche offers FlashText, Lunr.py retrievers that are not available in Haystack and that I needed for the project I wanted to solve. Haystack is clearly more complete but I think also more complex to use.

For the text part not requiring regex, https://github.com/vi3k6i5/flashtext might help

Spacy for parsing the Headlines, remove stop words etc. might be ok but I think the problem is quite narrow so a set of fixed regex searches might work quite well. If regex is too slow, try: https://github.com/vi3k6i5/flashtext

I would probably use something like [flashtext](https://github.com/vi3k6i5/flashtext) which should not be too hard to port to kotlin.

This is our optimization problem. Now, we hope that you have an idea of what our goal is. Luckily for us, this is already implemented in a Python module called gensim. Yes, these guys are brilliant in natural language processing and we will make use of it. 🤝

Try LDA as implemented in gemsin https://github.com/RaRe-Technologies/gensim

Standout python NLP libraries include Spacy and Gensim, as well as pre-trained model availability in Hugginface. These libraries have widespread use in and support from industry and it shows. Spacy has best-in-class methods for pre-processing text for further applications. Gensim helps you manage your corpus of documents, and contains a lot of different tools for solving a common industry task, topic modeling.

Here we have to install the gensim library in a jupyter notebook to be able to use it in our project, consider the code below;

Nice work! I wonder if you're going the same challenges that gensim had for being generic in summarization.

For context:

> Despite its general-sounding name, the module will not satisfy the majority of use cases in production and is likely to waste people's time.

https://github.com/RaRe-Technologies/gensim/wiki/Migrating-f...

TextRank will work without any problems. https://radimrehurek.com/gensim/

For the topic modelling itself, I am going to use Gensim library by Radim Rehurek, which is very developer friendly and easy to use.