Top 23 text-mining Open-Source Projects

• awesome-nlp

  3 15,977 4.3

  :book: A curated list of resources dedicated to Natural Language Processing (NLP)

• textract

  4 3,778 3.7 HTML

  extract text from any document. no muss. no fuss.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• texthero

  1 2,864 4.5 Python

  Text preprocessing, representation and visualization from zero to hero.

• trafilatura

  13 2,778 8.7 Python

  Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Project mention: Trafilatura: Python tool to gather text on the Web | news.ycombinator.com | 2023-08-14

The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features

Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.

• scattertext

  3 2,197 4.7 Python

  Beautiful visualizations of how language differs among document types.

• DataScienceR

  1 1,959 0.0 R

  a curated list of R tutorials for Data Science, NLP and Machine Learning

• tidytext

  3 1,158 3.2 R

  Text mining using tidy tools :sparkles::page_facing_up::sparkles:

Project mention: Erik Prince, Founder of Blackwater, Faces Indictment in Austria for Trafficking Arms to Libya in Violation of UN Arms Embargo | /r/law | 2023-06-17

Others have explored Trump’s timeline and noticed this tends to hold up- and Trump himself does indeed tweet from a Samsung Galaxy. But how could we examine it quantitatively? I’ve been writing about text mining and sentiment analysis recently, particularly during my development of the tidytext R package with Julia Silge, and this is a great opportunity to apply it again.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• rake-nltk

  4 1,034 0.0 Python

  Python implementation of the Rapid Automatic Keyword Extraction algorithm using NLTK.

• open-semantic-search

  4 905 3.3 Shell

  Open Source research tool to search, browse, analyze and explore large document collections by Semantic Search Engine and Open Source Text Mining & Text Analytics platform (Integrates ETL for document processing, OCR for images & PDF, named entity recognition for persons, organizations & locations, metadata management by thesaurus & ontologies, search user interface & search apps for fulltext search, faceted search & knowledge graph)

Project mention: Need a better way to organize documents in massive nas | /r/DataHoarder | 2023-07-06

documents, pdf's - https://opensemanticsearch.org

• LDAvis

  2 551 0.0 JavaScript

  R package for web-based interactive topic model visualization.

• awesome-sentiment-analysis

  1 526 1.9

  Repository with all what is necessary for sentiment analysis and related areas

• awesome-computational-social-science

  1 458 6.4 R

  A list of awesome resources for Computational Social Science

  

• awesome-bioie

  1 300 2.1

  🧫 A curated list of resources relevant to doing Biomedical Information Extraction (including BioNLP)

Project mention: Snomed CT Entity Linking Challenge | news.ycombinator.com | 2023-12-22

> The objective of this competition is to link spans of text in clinical notes with specific topics in the SNOMED CT clinical terminology. Participants will train models based on real-world doctor's notes which have been de-identified and annotated with SNOMED CT concepts by medically trained professionals. This is the largest publicly available dataset of labelled clinical notes, and you can be one of the first to use it!

NER: Named Entity Recognition: https://en.wikipedia.org/wiki/Named-entity_recognition

awsome-medical-coding-nlp: https://github.com/acadTags/Awesome-medical-coding-NLP

awesome-ehr-deep-learning: https://github.com/hurcy/awesome-ehr-deeplearning

awesome-ner: https://github.com/smiyawaki0820/awesome-ner

awesome-bioie > Research groups: https://github.com/caufieldjh/awesome-bioie#groups-active-in...

SNOMED-CT as RDF: https://sphn-semantic-framework.readthedocs.io/en/latest/ext...

• awesome-hungarian-nlp

  3 205 3.2

  A curated list of NLP resources for Hungarian

• converse

  6 176 0.0 Jupyter Notebook

  Conversational text Analysis using various NLP techniques

  

• textfeatures

  1 167 0.0 R

  👷‍♂️ A simple package for extracting useful features from character objects 👷‍♀️

• extractnet

  2 162 0.0 HTML

  A fork of Dragnet that also extract author, headline, date, keywords from context, as well as built in metadata extraction all in one package

  

• huspacy

  3 147 8.6 Python

  HuSpaCy: industrial-strength Hungarian natural language processing

  

• R-text-data

  1 140 0.0

  List of textual data sources to be used for text mining in R

• trrex

  2 134 3.5 Python

  Efficient string matching with regular expressions (by mesejo)

• orange3-text

  1 124 8.5 Python

  🍊 :page_facing_up: Text Mining add-on for Orange3

  

Project mention: Ask HN: What Underrated Open Source Project Deserves More Recognition? | news.ycombinator.com | 2024-03-07

• intertext

  1 110 0.0 Python

  Detect and visualize text reuse

  

Project mention: Ask HN: Tool to find text reuse, similar paragraphs, fuzzy/near dupes in folder? | news.ycombinator.com | 2023-06-25

Do you know of any too that I can use to compare my own notes and documents vault in search for copied paragraphs or almost similar phrases? Normal diffing/hashing wouldn't work as we're talking about the contents of slightly modified documents, and the comparison of each file against all others.

I found the following tools that seem related yet not quite there, maybe I'm missing a particular term of art?

https://github.com/YaleDHLab/intertext

• odinson

  3 66 4.5 Scala

  Odinson is a powerful and highly optimized open-source framework for rule-based information extraction. Odinson couples a simple, yet powerful pattern language that can operate over multiple representations of text, with a runtime system that operates in near real time.

  

Project mention: [D] Finetuning for text extraction (e.g. scientific sources) | /r/MachineLearning | 2023-06-11

Odinson

• SaaSHub

  www.saashub.com sponsored

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

text-mining related posts

• Erik Prince, Founder of Blackwater, Faces Indictment in Austria for Trafficking Arms to Libya in Violation of UN Arms Embargo
  1 project | /r/law | 17 Jun 2023
• [D] Finetuning for text extraction (e.g. scientific sources)
  1 project | /r/MachineLearning | 11 Jun 2023
• Szoláris Magyar: Az elmúlt négy hónapban egy morfológia alapú alternatív írásrendszeren dolgoztam, ami a magyar nyelvre illeszkedik (további infó kommentekben)
  1 project | /r/hungary | 22 Feb 2023
• [Q] Does anyone use R to code qualitative data?
  3 projects | /r/rstats | 16 Oct 2022
• Stream processing. The logic is encoded in the input pattern.
  1 project | /r/java | 25 Jul 2022
• Stream processing. The logic is encoded in the input pattern.
  1 project | /r/programming | 24 Jul 2022
• How to give a file path to a file parser when you only have an HTTPRequest?
  1 project | /r/learnpython | 13 Jan 2022
• A note from our sponsor - WorkOS
  workos.com | 25 Apr 2024

  The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning. Learn more →

Index

Sponsored

SaaSHub - Software Alternatives and Reviews

SaaSHub helps you find the best software and product alternatives

www.saashub.com