gensim VS pyffs

This is our optimization problem. Now, we hope that you have an idea of what our goal is. Luckily for us, this is already implemented in a Python module called gensim. Yes, these guys are brilliant in natural language processing and we will make use of it. 🤝

Try LDA as implemented in gemsin https://github.com/RaRe-Technologies/gensim

Standout python NLP libraries include Spacy and Gensim, as well as pre-trained model availability in Hugginface. These libraries have widespread use in and support from industry and it shows. Spacy has best-in-class methods for pre-processing text for further applications. Gensim helps you manage your corpus of documents, and contains a lot of different tools for solving a common industry task, topic modeling.

Here we have to install the gensim library in a jupyter notebook to be able to use it in our project, consider the code below;

Nice work! I wonder if you're going the same challenges that gensim had for being generic in summarization.

For context:

> Despite its general-sounding name, the module will not satisfy the majority of use cases in production and is likely to waste people's time.

https://github.com/RaRe-Technologies/gensim/wiki/Migrating-f...

TextRank will work without any problems. https://radimrehurek.com/gensim/

For the topic modelling itself, I am going to use Gensim library by Radim Rehurek, which is very developer friendly and easy to use.

Dramatic post ;) It'd be interesting to see concrete benchmarks, on some public data.

Btw I didn't find the Schulz & Mihov paper that cryptic. You can check its implementation in Python [0], pretty straightforward IMO.

But I should note that in the end, we chose a simpler approach: the FastSS index. It bypasses constructing / intersecting Levenshtein automata altogether, and is super fast [1].

[0] https://github.com/antoinewdg/pyffs

[1] Boytsov, Leonid. (2011). Indexing methods for approximate dictionary searching: Comparative analysis. http://boytsov.info/pubs/sisap2012.pdf