scrapy-playwright VS scrapy-rotating-proxies

Middleware libraries are written by the community and are extending their functionality. For example, scrapy-playwright.

While libraries like Scrapy require additional installation of middleware, i.e, scrapy-playwright and still doesn’t work with Windows, Crawlee for Python supports a unified interface for HTTP & headless browsers.

Scrapy does not support headless browsers natively, but it supports them with its plugin system, similarly it does not support scraping JavaScript rendered websites, but the plugin system makes this possible. One of the best examples is its Playwright plugin.

scrapy-playwright is an integration between Scrapy and Playwright. It enables scraping dynamic web pages with Scrapy by processing the web scraping requests using a Playwright instance.

To integrate Playwright with Scrapy, we will use the scrapy-playwright library. Then, we will scrape https://www.mintmobile.com/product/google-pixel-7-pro-bundle/ to demonstrate how to extract data from a website using Playwright and Scrapy.

Scrapy-playwright

your website is a dynamic there is many integration on scrappy can help you This the best best one https://github.com/scrapy-plugins/scrapy-playwright

This concern should be lifted if you are a Scrapy lover. There is a Scrapy integration for playwright, that gives you a lot of freedom and lets you operate from a Scrapy spider.

It was the sliding CAPTCHA but I solved it by following the instructions from the library I'm using to rotate proxies to retry with a different IP when there is a CAPTCHA https://github.com/TeamHG-Memex/scrapy-rotating-proxies At the bottom if anyone is interested

Hi, I've been using the scrapy-rotating-proxies (https://github.com/TeamHG-Memex/scrapy-rotating-proxies) library for scrapy and I'm getting logs in my crawl of type example: "[rotating_proxies.expire] DEBUG: Proxy is DEAD. When I check and test the proxies (I'm using webshare proxies) and urls mentioned on the logs individually they work ok, so I assume it's a problem with the library, has anyone had the same issue of similar problem? (I looked for tickets reported on github but had didn't find any refering to this.

Scrapy takes the proxy from the http_proxy/https_proxy env vars. They can include the user/password. As for pools, Scrapy itself doesn't support that, but you can use https://github.com/TeamHG-Memex/scrapy-rotating-proxies or similar addons to use them.

Hello, I'm looking into trying free proxies using something like in this github (https://github.com/TeamHG-Memex/scrapy-rotating-proxies/blob/master/README.rst). However, I need to find my own list of proxy IP's to use. When I look up free proxies I find plenty of options, but I'm rather new to this topic and don't know what to use. There seems to be plenty of different types, and I'm not sure if I should/shouldn't use certain proxy IP's. Any advice on the topic would be appreciated.