autoscraper VS scrapingant-client-python

See if something like autoscraper or mlscraper suits your needs.

GitHub - alirezamika/autoscraper: A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Not a library but a decent program to not reinvent the wheel I'm currently adding regular selector lookups back into it https://github.com/alirezamika/autoscraper

It depends on the website, but autoscraper was used to calculate similar nodes given the text to search. Not sure how it works now but it's open source.

Should check out this for method 1.

In this article, we will learn how to create a simple e-commerce search API with multiple platform support: eBay and Amazon. AutoScraper and FastAPi provide the ability to create a powerful JSON API for the date. With Playwright's help, we'll extend our scraper and avoid blocking by using ScrapingAnt's web scraping API.

Maybe you could try out this tool never used it myself https://github.com/alirezamika/autoscraper Or you would need to train a neural network which does the parsing for you but the accuracy won't be 100% with this method https://github.com/scrapy/scrapely To answer your question there is no way to write one program which scrapes different pages correctly because their structure is not the same and some of them probably will have protection which will block your scraper so that would need extra attention or additional code

Hello. In order to our previous conversation: We've created a python client library for our API: https://github.com/ScrapingAnt/scrapingant-client-python So you can check it out. Scrapy plugin is still under development. I'll keep you posted. Thanks for the interest :-)