wtf_wikipedia
mlscraper
Our great sponsors
| wtf_wikipedia | mlscraper | |
|---|---|---|
| 1 | 10 | |
| 743 | 1,225 | |
| - | - | |
| 8.0 | 0.6 | |
| 13 days ago | about 1 month ago | |
| JavaScript | Python | |
| MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
wtf_wikipedia
-
Experimental library for scraping websites using OpenAI's GPT API
This may finally be a solution for scraping wikipedia and turning it into structured data. (Or do we even need structured data in the post-AI age?)
Mediawiki is notorious for being hard to parse:
* https://github.com/spencermountain/wtf_wikipedia#ok-first- - why it's hard
* https://techblog.wikimedia.org/2022/04/26/what-it-takes-to-p... - an entire article about parsing page TITLES
* https://osr.cs.fau.de/wp-content/uploads/2017/09/wikitext-pa... - a paper published about a wikitext parser
mlscraper
-
What are the best tools for web scraping and analysis of natural language to populate a dataset?
See if something like autoscraper or mlscraper suits your needs.
-
Experimental library for scraping websites using OpenAI's GPT API
Why GPT-based then? There are libraries that do this: You give examples, they generate the rules for you and give you a scraper object that takes any html and returns the scraped data.
Mine: https://github.com/lorey/mlscraper
-
Could someone recommend me a library for c# like one of these two (they are for python) : mlscraper and autoscraper
GitHub - lorey/mlscraper: 🤖 Scrape data from HTML websites automatically by just providing examples
-
Smart Scraper
Check it out here: https://github.com/lorey/mlscraper Example: https://github.com/lorey/mlscraper/blob/master/examples/quotes\_to\_scrape.py
- Pre-trained Webscraping Models
- 🤖 Scrape data from HTML websites automatically by just providing examples
- mlscraper: Scrape data from HTML pages automatically with Machine Learning
-
Show HN: RSS feeds for arbitrary websites using CSS selectors
In case anyone wants to detect the selectors automatically, here's a small python library I wrote that does it for you: https://github.com/lorey/mlscraper
What are some alternatives?
sdow - Six Degrees of Wikipedia
scrapingant-client-python - ScrapingAnt API client for Python.
anon - tweet about anonymous Wikipedia edits from particular IP address ranges
ttrss_plugin-feediron - Evolution of ttrss_plugin-af_feedmod
duckling - Language, engine, and tooling for expressing, testing, and evaluating composable language rules on input strings.
furss - Fix Up RSS (and atom): Make full-text versions of rss/atom feeds
autoscraper - A Smart, Automatic, Fast and Lightweight Web Scraper for Python
feed-me-up-scotty
scrapeghost - 👻 Experimental library for scraping websites using OpenAI's GPT API.
rssify - Tool that generates an rss feed out of websites that don't have one
RSSHub - 🧡 Everything is RSSible
feedgen - Generates RSS/ATOM/JSON feeds. Can be reasonably extended or create a feed using the CSS generator.