corex_topic
gensim
corex_topic | gensim | |
---|---|---|
5 | 18 | |
622 | 15,256 | |
- | 0.9% | |
3.8 | 7.5 | |
about 3 years ago | 11 days ago | |
Python | Python | |
Apache License 2.0 | GNU Lesser General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
corex_topic
-
[NLP] Topic Identification & Quantisation
We ended up settling on a HuggingFace transformer + HDBSCAN pipeline from BERTopic. I like this because it makes it straightforward to tune and test, and you probabilistically assign documents to clusters, so you can do interesting aggregation and sampling after you have your inference done, like selecting text. Other options include top2vec which basically does the same thing without some guiding tools available in BERTopic. Either is suitable for what you’re doing. Older techniques include things like the Latent Dirichlet Allocation and COREX.
-
NLP: How to visualise the main context (in the form of words, sentences etc) of a text document?
If I understand correctly, it sounds like you are trying to do Topic Modeling on documents. One python library I’ve enjoyed using is https://github.com/gregversteeg/corex_topic which, besides doing general topic modeling, also learns hierarchies of topics.
-
Are topic models reliable or useful?
We started off by trying LDA and NMF, but the topics were too messy so we wound up switching to CorEx (https://github.com/gregversteeg/corex_topic), which is a semi-supervised algo that lets you "nudge" the model in the right direction using anchor terms. By the time our topics started looking coherent, it turned out that a regex with the anchor terms we'd picked outperformed the model itself. This case study was on a relatively small sample of relatively short documents (~4k survey open-ends) but for what it's worth, we also tried to use topic models to classify congressional Facebook posts (much larger corpus and longer documents) and the results were the same.
Overfitting is certainly part of the problem - in one of my earlier posts I talk about "conceptually spurious words," which are essentially the product of overfitting - but the more difficult problem is polysemy. I'm sure there are ways to mitigate that - expanding the feature space with POS tagging, etc. - but ultimately I think the solution is to simply avoid using a dimensionality reduction method for text classification. Supervised models are clearly the way to go - even if those "models" are just keyword dictionaries curated based on domain knowledge.
-
Trying to read text documents and allow for up to m labels per documents, like suggested tags, but the number of labels can be different for each document. Any advice?
Unsupervised is also possible for topic modelling: CorEX
-
NLP Problem
You could use CorEX for topic modelling. With that you can find the type of causes and from there their distribution. Then you can join it with other non-textual meta-data.
gensim
- Aggregating news from different sources
-
Understanding How Dynamic node2vec Works on Streaming Data
This is our optimization problem. Now, we hope that you have an idea of what our goal is. Luckily for us, this is already implemented in a Python module called gensim. Yes, these guys are brilliant in natural language processing and we will make use of it. 🤝
-
Topic modeling --- allow multiple topics per statement
Try LDA as implemented in gemsin https://github.com/RaRe-Technologies/gensim
-
Is it home bias or is data wrangling for machine learning in python much less intuitive and much more burdensome than in R?
Standout python NLP libraries include Spacy and Gensim, as well as pre-trained model availability in Hugginface. These libraries have widespread use in and support from industry and it shows. Spacy has best-in-class methods for pre-processing text for further applications. Gensim helps you manage your corpus of documents, and contains a lot of different tools for solving a common industry task, topic modeling.
- sentence transformer vector dimensionality reduction to 1
- Where to start for recommendation systems
-
GET STARTED WITH TOPIC MODELLING USING GENSIM IN NLP
Here we have to install the gensim library in a jupyter notebook to be able to use it in our project, consider the code below;
-
Show HN: I built a site that summarizes articles and PDFs using NLP
Nice work! I wonder if you're going the same challenges that gensim had for being generic in summarization.
For context:
> Despite its general-sounding name, the module will not satisfy the majority of use cases in production and is likely to waste people's time.
https://github.com/RaRe-Technologies/gensim/wiki/Migrating-f...
-
[Research] Text summarization using Python, that can run on Android devices?
TextRank will work without any problems. https://radimrehurek.com/gensim/
-
Topic modelling with Gensim and SpaCy on startup news
For the topic modelling itself, I am going to use Gensim library by Radim Rehurek, which is very developer friendly and easy to use.
What are some alternatives?
GuidedLDA - semi supervised guided topic model with custom guidedLDA
BERTopic - Leveraging BERT and c-TF-IDF to create easily interpretable topics.
ennemi - Easy Nearest Neighbor Estimation of Mutual Information
scikit-learn - scikit-learn: machine learning in Python
MLflow - Open source platform for the machine learning lifecycle
tensorflow - An Open Source Machine Learning Framework for Everyone
Keras - Deep Learning for humans
flair - A very simple framework for state-of-the-art Natural Language Processing (NLP)
fuzzywuzzy - Fuzzy String Matching in Python
xgboost - Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library, for Python, R, Java, Scala, C++ and more. Runs on single machine, Hadoop, Spark, Dask, Flink and DataFlow
CNTK - Microsoft Cognitive Toolkit (CNTK), an open source deep-learning toolkit