Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality. Learn more →
Top 8 Python Corpu Projects
-
trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
japanese-words-to-vectors
Word2vec (word to vectors) approach for Japanese language using Gensim and Mecab.
-
open-discourse
Open Discourse is the first fully comprehensive corpus of the plenary proceedings of the federal German Parliament (Bundestag).
-
open-australian-legal-corpus-creator
The code used to create and update the Open Australian Legal Corpus, the first and only multijurisdictional open corpus of Australian legislative and judicial documents.
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
korean-word-ipa-dictionary
Dictionary of pairs of Korean word and IPA crawled from Wiktionary (Korean edition)
Project mention: Trafilatura: Python tool to gather text on the Web | news.ycombinator.com | 2023-08-14The feature list answers that question pretty well: https://github.com/adbar/trafilatura#features
Basically: you could implement all of this on top of BeautifulSoup - polite crawling policies, sitemap and feed parsing, URL de-duplication, parallel processing, download queues, heuristics for extracting just the main article content, metadata extraction, language detection... but it would require writing an enormous amount of extra code.
Project mention: Show HN: New AI Dataset Based on LibGen and Sci-Hub | news.ycombinator.com | 2023-09-08
CC BY SA 3.0: https://github.com/amir-zeldes/gum/blob/master/LICENSE.txt
I didn't know about that project, that's really cool! I'd be curious to know whether the person who devised this scheme was aware of structured meaning representations (UCCA, AMR, ...), and if so, why they chose to create a new meaning representation. Maybe the goals of the project and/or the constraints of Wikidata necessitated this.
Anyway, GUM (and its sister corpus EWT) does have a lot of parsed permissively-licensed text, so whoever's in charge should definitely consider using them. (Amir, the maintainer, is also super friendly and would respond to an email.)
Project mention: Show HN: Mapping almost every law, regulation and case in Australia | news.ycombinator.com | 2024-03-22Hey HN,
After months of hard work, I am excited to share the first ever semantic map of Australian law.
My map represents the first attempt to map Australian laws, cases and regulations across the Commonwealth, States and Territories semantically, that is, by their underlying meaning.
Each point on the map is a unique document in the [Open Australian Legal Corpus](https://huggingface.co/datasets/umarbutler/open-australian-l...), the largest open database of Australian law (which, full disclosure, I [created](https://umarbutler.com/how-i-built-the-largest-open-database...)). The closer any two points are on the map, the more similar they are in underlying meaning.
As I cover in my article, there’s a lot you can learn by mapping Australian law. Some of the most interesting insights to come out of this initiative are that:
⦁ Migration, family and substantive criminal law are the most isolated branches of case law on the map;
⦁ Migration, family and substantive criminal law are the most distant branches of case law from legislation on the map;
⦁ Development law is the closest branch of case law to legislation on the map;
⦁ Case law is more of a continuum than a rigidly defined structure and the borders between branches of case law can often be quite porous; and
⦁ The map does not reveal any noticeable distinctions between Australian state and federal law, whether it be in style, principles of interpretation or general jurisprudence.
If you’re interested in learning more about what the map has to teach us about Australian law or if you’d like to find out how you can create semantic maps of your own, check out the full article on my blog, which provides a detailed analysis of my map and also covers the finer details of how I built it, with code examples offered along the way.
NOTE:
The open source projects on this list are ordered by number of github stars.
The number of mentions indicates repo mentiontions in the last 12 Months or
since we started tracking (Dec 2020).
Python Corpus related posts
- Show HN: Mapping almost every law, regulation and case in Australia
- Évariste Galois
- Show HN: New AI Dataset Based on LibGen and Sci-Hub
- Obtaining a Word List
- Can chat GPT overtake Google if they play their cards right?
- Mechanical engineering professor seeking recommendations for foundational textbooks or seminal publications (corpus ling., genre analysis)
- English-conversation corpus
-
A note from our sponsor - InfluxDB
www.influxdata.com | 27 Apr 2024
Index
What are some of the best open-source Corpu projects in Python? This list will help you:
| Project | Stars | |
|---|---|---|
| 1 | trafilatura | 2,778 |
| 2 | bookcorpus | 778 |
| 3 | Korpora | 645 |
| 4 | gum | 86 |
| 5 | japanese-words-to-vectors | 83 |
| 6 | open-discourse | 81 |
| 7 | open-australian-legal-corpus-creator | 56 |
| 8 | korean-word-ipa-dictionary | 18 |
Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com