scrapeghost VS mlscraper

Not my project but an ex-colleague has been having some success in this direction: https://jamesturk.github.io/scrapeghost/

Yes, there is something like that available - ScrapeGhost.

With AI scraping, tools can be far more resilient than soon enough to minor dom changes. See - https://jamesturk.github.io/scrapeghost/.

Their ToS mentions scraping but it pertains to scraping their frontend instead of using their API, which they don't want you to do.

Also - this library requests the HTML by itself [0] and ships it as a prompt but with preset system messages as the instruction [1].

[0] - https://github.com/jamesturk/scrapeghost/blob/main/src/scrap...

[1] - https://github.com/jamesturk/scrapeghost/blob/main/src/scrap...

See if something like autoscraper or mlscraper suits your needs.

Why GPT-based then? There are libraries that do this: You give examples, they generate the rules for you and give you a scraper object that takes any html and returns the scraped data.

Mine: https://github.com/lorey/mlscraper

GitHub - lorey/mlscraper: 🤖 Scrape data from HTML websites automatically by just providing examples

Check it out here: https://github.com/lorey/mlscraper Example: https://github.com/lorey/mlscraper/blob/master/examples/quotes\_to\_scrape.py

In case anyone wants to detect the selectors automatically, here's a small python library I wrote that does it for you: https://github.com/lorey/mlscraper