pyppeteer VS playwright-python

{ "issues":[ { "date":"3 weeks ago", "title":"can not get document object in evaluate function", "author":"datuizhuang", "description":"User is unable to get document object in evaluate function, receiving None as result. When using page.evaluate(\\'document\\') or page.evaluate(\"document.querySelector(\\'body\\')\") it returns None, but page.querySelector(\\'body\\') works correctly." "link": "https://github.com/pyppeteer/pyppeteer/issues/486" }, { "date":"2 weeks ago", "title":"Generate fingerprint", "author":"krolaper", "description":"Issue #487 related to fingerprint generation, though specific details are not provided in the given text.", "link": "https://github.com/pyppeteer/pyppeteer/issues/487" } ] }

The latest version of Pyppeteer, i.e., 1.0.2, can also be installed by executing pip3 install -U git+https://github.com/pyppeteer/pyppeteer@dev on the terminal.

Having async baked into Python both as a keyword and standard library has vastly opened up the number of use cases async will be used in where it is appropriate, one of my favorite libraries pyppeteer would have not existed in the easy nice way it can be used without asyncio: https://github.com/pyppeteer/pyppeteer

Yes, this is scraping. You need a legit new cookie issued by the site. The solution is to automate the going to get the cookies part. The most straight forward way to do this is probably using a headless browser through something like Selenium or pyppeteer.

Last I checked, pyppeteer wasn't a thing anymore, and I haven't tried Playwright but if it has a headless mode, thats what you want so you don't have a browser open.

pip install -U git+https://github.com/pyppeteer/pyppeteer@dev

The normal Puppeteer package is JavaScript, but I do see that there's a Python port called pyppeteer. I can't vouch for it specifically, but I imagine it's similarly easy to use as the JS version.

TikTok uses quite a lot of JavaScript on its site, both for displaying content and for analyzing user behavior, including detecting and blocking crawlers. Therefore, for crawling TikTok, we'll use a headless browser with Playwright.

Integrating Ollama, Microsoft vision models and Playwright I've made a simple agent that can browse websites and data to answer your query.

Do you have to use Curl? It wouldn't render a lot of sites correctly anyway (anything that uses JS for rendering).

Can you run a puppeteer/playwright instance and add an ad blocker to that? e.g. https://github.com/ghostery/adblocker or https://github.com/microsoft/playwright-python/issues/782

Playwright

An alternative is to use a package like playwright (or Selenium) to run a browser remotely and login.

Using the tools at hand is often the best approach. That said, I've spent most of the last 13 years of my career automating browsers. For years, I used Selenium with a variety of libraries. After switching to Puppeteer/Playwright, I have zero interest in going back lol. Playwright actually has first party Python support. (Puppeteer has a port called Pyppeteer, but it's no longer maintained and the author recommends using Playwright)

https://playwright.dev/python/

This is pretty great for web automation https://playwright.dev/python/