SaaSHub helps you find the best software and product alternatives Learn more →
Top 22 Corpu Open-Source Projects
-
corpora
A collection of small corpuses of interesting data for the creation of bots and similar stuff.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
-
-
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
-
japanese-words-to-vectors
Word2vec (word to vectors) approach for Japanese language using Gensim and Mecab.
-
open-discourse
Open Discourse is the first fully comprehensive corpus of the plenary proceedings of the federal German Parliament (Bundestag).
-
open-australian-legal-corpus-creator
The code used to create and update the Open Australian Legal Corpus, the first and only multijurisdictional open corpus of Australian legislative and judicial documents.
-
shabby-pages
ShabbyPages is a state-of-the-art corpus of born-digital document images with both ground truth and distorted versions appropriate for use in training models to reverse distortions and recover to original denoised documents.
-
-
contract-discovery
Data and additional information regarding the paper: Contract Discovery. Dataset and a Few-Shot Semantic Retrieval Challenge with Competitive Baselines (to appear in Findings of EMNLP).
-
korean-word-ipa-dictionary
Dictionary of pairs of Korean word and IPA crawled from Wiktionary (Korean edition)
-
Romanian-Word-Embeddings
Romanian Word Embeddings. Here you can find pre-trained corpora of word embeddings. Current methods: CBOW, Skip-Gram, Fast-Text (from Gensim library). The .vec and .model files are available for download (all in one archive).
-
katya-dev
Katya or The Liberated Corpus a text corpus that allows you to request and scrape any web resource!
-
-
SaaSHub
SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives
Project mention: Corpora: A collection of small corpuses of interesting data | news.ycombinator.com | 2024-03-25
Project mention: Trafilatura: Python tool to gather text on the Web | news.ycombinator.com | 2023-08-14The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features
Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.
Project mention: Show HN: New AI Dataset Based on LibGen and Sci-Hub | news.ycombinator.com | 2023-09-08
CC BY SA 3.0: https://github.com/amir-zeldes/gum/blob/master/LICENSE.txt
I didn't know about that project, that's really cool! I'd be curious to know whether the person who devised this scheme was aware of structured meaning representations (UCCA, AMR, ...), and if so, why they chose to create a new meaning representation. Maybe the goals of the project and/or the constraints of Wikidata necessitated this.
Anyway, GUM (and its sister corpus EWT) does have a lot of parsed permissively-licensed text, so whoever's in charge should definitely consider using them. (Amir, the maintainer, is also super friendly and would respond to an email.)
Project mention: Show HN: Mapping almost every law, regulation and case in Australia | news.ycombinator.com | 2024-03-22Hey HN,
After months of hard work, I am excited to share the first ever semantic map of Australian law.
My map represents the first attempt to map Australian laws, cases and regulations across the Commonwealth, States and Territories semantically, that is, by their underlying meaning.
Each point on the map is a unique document in the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-l...), the largest open database of Australian law (which, full disclosure, I [created](https://umarbutler.com/how-i-built-the-largest-open-database...)). The closer any two points are on the map, the more similar they are in underlying meaning.
As I cover in my article, there’s a lot you can learn by mapping Australian law. Some of the most interesting insights to come out of this initiative are that:
⦁ Migration, family and substantive criminal law are the most isolated branches of case law on the map;
⦁ Migration, family and substantive criminal law are the most distant branches of case law from legislation on the map;
⦁ Development law is the closest branch of case law to legislation on the map;
⦁ Case law is more of a continuum than a rigidly defined structure and the borders between branches of case law can often be quite porous; and
⦁ The map does not reveal any noticeable distinctions between Australian state and federal law, whether it be in style, principles of interpretation or general jurisprudence.
If you’re interested in learning more about what the map has to teach us about Australian law or if you’d like to find out how you can create semantic maps of your own, check out the full article on my blog, which provides a detailed analysis of my map and also covers the finer details of how I built it, with code examples offered along the way.
Project mention: Building a ChatGPT-like AI for Bangla. Please suggest some places where Bangla text data is plentiful. | /r/bangladesh | 2023-06-07Check this repo of Bangla corpus, also Banglapedia & Wikipedia Bangla.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Corpus related posts
-
Corpora: A collection of small corpuses of interesting data
-
Show HN: Mapping almost every law, regulation and case in Australia
-
Évariste Galois
-
I built the largest open database of Australian law
-
Show HN: New AI Dataset Based on LibGen and Sci-Hub
-
Obtaining a Word List
-
Can chat GPT overtake Google if they play their cards right?
-
A note from our sponsor - SaaSHub
www.saashub.com | 4 May 2024
Index
What are some of the best open-source Corpu projects? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | corpora | 4,851 |
| 2 | Chinese-Names-Corpus | 3,829 |
| 3 | trafilatura | 2,853 |
| 4 | quanteda | 824 |
| 5 | bookcorpus | 778 |
| 6 | Korpora | 645 |
| 7 | ua-gec | 254 |
| 8 | awesome-hungarian-nlp | 207 |
| 9 | wp2txt | 168 |
| 10 | bible-corpus | 163 |
| 11 | gum | 86 |
| 12 | japanese-words-to-vectors | 83 |
| 13 | open-discourse | 81 |
| 14 | open-australian-legal-corpus-creator | 57 |
| 15 | shabby-pages | 42 |
| 16 | proiel-treebank | 33 |
| 17 | contract-discovery | 25 |
| 18 | korean-word-ipa-dictionary | 18 |
| 19 | Romanian-Word-Embeddings | 12 |
| 20 | bangla-corpus | 11 |
| 21 | katya-dev | 5 |
| 22 | empirist-corpus | 2 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com