MechanicalSoup VS crawlee

Language: Python | GitHub: 4.7K+ stars | link

MechanicalSoup is a Python library for web scraping that combines the simplicity of Requests with the convenience of BeautifulSoup. It's particularly useful for interacting with web forms, like login pages. Here's a basic example to illustrate how you can use MechanicalSoup for web scraping:

Try with Mechanicalsoup https://mechanicalsoup.readthedocs.io/en/stable/

You should try MechanicalSoup it uses Beautifulsoup but provides a simpler API using its StatefulBrowser filling forms and doing some other stuff is easier than just directly using requests and Beautifulsoup.

MechanicalSoup

If you're a fan of Playwright check out Crawlee [0]. I've used it for a few small projects and it's been faster for me to get what I've needed done.

[0] https://crawlee.dev/

uvx crawlee['cli'] create tiktok-crawlee --crawler-type playwright --http-client httpx --package-manager uv --apify --start-url 'https://crawlee.dev'

View on GitHub

:::tip If you like the blog so far, please consider giving Crawlee a star on GitHub, it helps us to reach and help more developers. :::

Check out Crawlee

Developed by Apify, it is a Python adaptation of their famous JS framework crawlee, first released on Jul 9, 2019.

The main advantage (for now) is that the library has a single interface for both HTTP and headless browsers, and bundled auto scaling. You can write your crawlers using the same base abstraction, and the framework takes care of this heavy lifting. Developers of scrapers shouldn't need to reinvent the wheel, and just focus on building the "business" logic of their scrapers. Having said that, if you wrote your own crawling library, the motivation to use Crawlee might be lower, and that's fair enough.

Please note that this is the first release, and we'll keep adding many more features as we go, including anti-blocking, adaptive crawling, etc. To see where this might go, check https://github.com/apify/crawlee

import asyncio from crawlee.playwright_crawler import PlaywrightCrawler, PlaywrightCrawlingContext async def main() -> None: # Create a crawler instance crawler = PlaywrightCrawler( # headless=False, # browser_type='firefox', ) @crawler.router.default_handler async def request_handler(context: PlaywrightCrawlingContext) -> None: data = { "request_url": context.request.url, "page_url": context.page.url, "page_title": await context.page.title(), "page_content": (await context.page.content())[:10000], } await context.push_data(data) await crawler.run(["https://crawlee.dev"]) if __name__ == "__main__": asyncio.run(main())

If you like reading this blog, we would be really happy if you gave Crawlee a star on GitHub!