scrapy-playwright VS YouTube.js

scrapy-playwright is an integration between Scrapy and Playwright. It enables scraping dynamic web pages with Scrapy by processing the web scraping requests using a Playwright instance.

To integrate Playwright with Scrapy, we will use the scrapy-playwright library. Then, we will scrape https://www.mintmobile.com/product/google-pixel-7-pro-bundle/ to demonstrate how to extract data from a website using Playwright and Scrapy.

Scrapy-playwright

your website is a dynamic there is many integration on scrappy can help you This the best best one https://github.com/scrapy-plugins/scrapy-playwright

This concern should be lifted if you are a Scrapy lover. There is a Scrapy integration for playwright, that gives you a lot of freedom and lets you operate from a Scrapy spider.

Now we need to modify scrapy's settings to allow it to work with playwright. Instructions can be found on playwright's GitHub page. We need to add settings for DOWNLOAD_HANDLERS and TWISTED_REACTOR. New settings that were added can be found between ###. This is what the settings file should look like:

You can use something like scrapy-playwright[0] to run a headless browser framework as your download handler. I think there are versions for some of the other headless systems, if you prefer those.

[0] https://github.com/scrapy-plugins/scrapy-playwright

[1]: https://github.com/scrapy-plugins/scrapy-playwright/issues/61

BTW, I don't understand the workaround: https://github.com/iv-org/invidious/pull/4552/files

Which was taken from here: https://github.com/LuanRT/YouTube.js/pull/624

Could anybody explain it to me?

There is a javascript library for interfacing with Youtube's API directly. It can also run on browsers. Using this, it's pretty easy to create a simple extension that replaces the default video player with your own. You can do a lot to improve your experience this way. I've made one which allows for higher quality streaming, pre-buffering video in the background, more subtitling options, etc... [2].

[1] https://github.com/LuanRT/YouTube.js

You can try Tod usw this youtubei.js

Love this approach. We can just bypass all the normal web scraping and get the structured data straight from the source. These APIs are usually no less stable than the ever changing HTML structure anyways.

Case study: YouTube.js

https://news.ycombinator.com/item?id=31021611

https://github.com/LuanRT/YouTube.js