autoscraper VS fastapi

See if something like autoscraper or mlscraper suits your needs.

GitHub - alirezamika/autoscraper: A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Not a library but a decent program to not reinvent the wheel I'm currently adding regular selector lookups back into it https://github.com/alirezamika/autoscraper

It depends on the website, but autoscraper was used to calculate similar nodes given the text to search. Not sure how it works now but it's open source.

Should check out this for method 1.

In this article, we will learn how to create a simple e-commerce search API with multiple platform support: eBay and Amazon. AutoScraper and FastAPi provide the ability to create a powerful JSON API for the date. With Playwright's help, we'll extend our scraper and avoid blocking by using ScrapingAnt's web scraping API.

Maybe you could try out this tool never used it myself https://github.com/alirezamika/autoscraper Or you would need to train a neural network which does the parsing for you but the accuracy won't be 100% with this method https://github.com/scrapy/scrapely To answer your question there is no way to write one program which scrapes different pages correctly because their structure is not the same and some of them probably will have protection which will block your scraper so that would need extra attention or additional code

That’s when I found FastAPI.

FastAPI & Uvicorn

Discussion at FastAPI GitHub: https://github.com/tiangolo/fastapi/issues/2696

An API application framework (such as FastAPI)

It’s been my experience that async Python frameworks tend to turn IO bound problems into CPU bound problems with a high enough request rate, because due to their nature they act as unbounded queues.

This ends up made worse if you’re using sync routes.

If you’re constrained on a resource such as a database connection pool, your framework will continue to pull http requests off the wire that a sane client will cancel and retry due to timeouts because it takes too long to get a connection out of the pool. Since there isn’t a straightforward way to cancel the execution of a route handler in every Python http framework I’ve seen exhibit this problem, the problem quickly snowballs.

This is an issue with fastapi, too- https://github.com/tiangolo/fastapi/issues/5759

Fast API.

We will create our API using FastAPI, a modern high-performance web framework for building fast APIs with Python. It is designed to be easy to use, efficient, and highly scalable. Some key features of FastAPI include:

FastAPI is a modern, fast, web framework for building APIs with Python 3.7+ based on standard Python type hints. It is designed to be easy to use, fast to run, and secure. In this blog post, we’ll explore the key features of FastAPI and walk through the process of creating a simple API using this powerful framework.

FastAPI is a modern, fast web framework for building APIs with Python 3.7+ that automatically generates OpenAPI and JSON Schema documentation. While FastAPI simplifies API development, manually creating and updating API documentation can still be a time-consuming task. In this blog post, we’ll explore how to leverage FastAPI’s automatic documentation generation capabilities, specifically focusing on Swagger and ReDoc, and how to streamline the process of documenting your APIs.