playwright-python VS mitmproxy

TikTok uses quite a lot of JavaScript on its site, both for displaying content and for analyzing user behavior, including detecting and blocking crawlers. Therefore, for crawling TikTok, we'll use a headless browser with Playwright.

Integrating Ollama, Microsoft vision models and Playwright I've made a simple agent that can browse websites and data to answer your query.

Do you have to use Curl? It wouldn't render a lot of sites correctly anyway (anything that uses JS for rendering).

Can you run a puppeteer/playwright instance and add an ad blocker to that? e.g. https://github.com/ghostery/adblocker or https://github.com/microsoft/playwright-python/issues/782

Playwright

An alternative is to use a package like playwright (or Selenium) to run a browser remotely and login.

Using the tools at hand is often the best approach. That said, I've spent most of the last 13 years of my career automating browsers. For years, I used Selenium with a variety of libraries. After switching to Puppeteer/Playwright, I have zero interest in going back lol. Playwright actually has first party Python support. (Puppeteer has a port called Pyppeteer, but it's no longer maintained and the author recommends using Playwright)

https://playwright.dev/python/

This is pretty great for web automation https://playwright.dev/python/

HTTP proxies like mitmproxy can intercept and export traffic as HAR

> if the rendering engine and network fetching were easily separable - and you could insert your own steps into that pipeline, you could do all sorts of neat stuff.

Can’t that be done relatively easily with https://mitmproxy.org/?

Enter mitmproxy - this beautiful Python program can act as a proxy and logs all network requests. It even comes with a devtool like web UI. Just what I needed.

Years ago, I set up https://mitmproxy.org on a Raspberry Pi and used it to get logs of every site that my kids would visit. I should be clear that monitoring/spying != parenting, but it definitely made me feel a little better to have some idea of what the kids are using the internet for.

From a technical perspective, it did exactly what you want. I had logs of full urls (not just domains). So, for example, I could view what they googled and when, if I wanted to anyway.

It did involve installing a certificate on the computer that they use, but there are how-to guides so setting everything up was simply a matter of following instructions.

The biggest drawback is that it noticeably slowed their internet. I imagine if I had run this on a more powerful computer it may have been better.

The bug issue was reproducible in the production setup, the logs/metrics were not so useful with the clues for the cause. So, I cloned the project code to my laptop and launched a Postgres instance via Docker Compose. Additionally, I started mitmproxy to be able to intercept and inspect HTTP requests on my machine, and created a template of the request to the Internal service API with my own SSN in Postman. My debugging setup was ready, so it was time to run and test the app.

So time to over-engineer this simple problem: since my gym uses EGym / Netpulse, it has Member Card NFC check-ins, which can be accessed via a private API that is called within their App. Using mitmproxy allowed me to quickly identify the check-in related endpoints and the auth mechanism.

Isn't that the point of mitmproxy? https://github.com/mitmproxy/mitmproxy