mini_racer VS Playwright

Some years ago I was on a shitty job - not technically, but the company turned out to be inhumane - at a Ruby shop, and on the side I was toying with mini_racer and I just upgraded to some macOS beta where it failed to build. A shitty +1-1 hack† for a compiler flag later and it was back flying.

A month later I received a cold email from a CTO to chat a bit about that PR, turns out they were using mini_racer heavily and forked it for their own purpose, and also created PyMiniRacer for the Python side of things. Next thing I know I got hired. Two years later the company got acquired.

Of course conditionally adding a compiler flag wasn't what got me hired per se, it only got my profile noticed. Probably side projects such as porting go by example to Ruby by implementing a ~1:1 CSP channel API[1], an Electron desktop client for Mattermost basically on a dare[2], ex mode for the Atom editor so that I could have that frackin' `:w`[3], leveraging Blocks to bolt on object-oriented-ness onto C because "closures are a poor man's object"[4], or reverse-engineering the Xbox One USB gamepad and writing a kext to turn it into a HID device on macOS from scratch on a lonely 7+h train ride with passengers judgementally staring at me sideways[4] probably contributed to it a bit.

My takeaway: luck is when preparation meets opportunity; but don't to side projects to get hired, because if you don't get hired then that time is lost. Rather, of all things, scratch your itch, have fun, embrace whatever quirkiness you fancy; no one can take that away from you.

[0]: https://github.com/rubyjs/mini_racer/commit/2086db1bbf2b5de4...

[1]: https://github.com/lloeki/normandy

[2]: https://github.com/lloeki/matterfront

[3]: https://github.com/lloeki/ex-mode

[4]: https://github.com/lloeki/cblocks-clobj/blob/master/main.c

[5]: https://github.com/lloeki/xbox_one_controller

Cue libv8-node+mini_racer from which PyMiniRacer was born. It is non-trivial but not as hard as one might think.

The most painful part is the libv8 build system and Google tooling, which makes it an absolute PITA for libv8 consumers that are not Chrome.

This is why the libv8 gem was atrocious to keep up to date and to build for several platforms, and why libv8-node was born, because the node build system and source distribution are actually sane.

Disclaimer: worked at Sqreen, now maintainer of libv8-node and collaborator of mini_racer

https://github.com/sqreen/PyMiniRacer

https://github.com/rubyjs/mini_racer

https://github.com/rubyjs/libv8-node

Eventually, we’ll want to call this JavaScript from a Rails controller using ExecJS or MiniRacer or some similar tool. None of the Ruby-to-JavaScript gems I found were sophisticated enough to know how to load npm modules with common.js or ES module syntax, so my solution is to just build the JavaScript with a watcher and have that run as part of bin/dev.

https://github.com/rubyjs/mini_racer/issues/190 has some info an then this PR: https://github.com/rubyjs/mini_racer/pull/210 that has been merged. Hope that all helps!

Playwright, an end-to-end test runner;

We start with a project that was bootstrapped with npx create-next-app. For the E2E test we use Playwright and set it up as described in the testing guide provided by Next.js.

Playwright is a powerful tool developed by Microsoft, it allows developers to write reliable end-to-end tests and perform browser automation tasks with ease. What sets Playwright apart is its ability to work seamlessly across multiple browsers (Chrome, Firefox, and WebKit), it provides a consistent and efficient way to interact with web pages, extract data, and automate repetitive tasks. Moreover, it supports various programming languages such as Node.js, Python, Java, and .NET, that’s making it a versatile choice for web scraping projects. Whether you're scraping public data for analysis, building a web crawler, or automating manual workflows, Playwright has you covered.

The consensus I could gather is either use playwright or use a workaround to solve it in the puppeteer layer. The root cause of the bug is a websocket size limitation on the CDP protocol for chromium.

With the advent of tools like Puppeteer and now Playwright, end-to-end testing has become much easier and more reliable. For anyone who's used Selenium in the past, you know what I'm talking about. Puppeteer has opened the way in terms of E2E tooling, but Playwright has taken it to the next level and made it easier to await for certain selectors or conditions to be fulfilled (via locators), thus making tests more reliable and less flaky. Also, it's a game changer that it introduced a test-runner - this made the integration between the headless browser and the actual test code much smoother.

In this tutorial, our main focus will be on Playwright web scraping. So what is Playwright? It’s a handy framework created by Microsoft. It's known for making web interactions more streamlined and works reliably with all the latest browsers like WebKit, Chromium, and Firefox. You can also run tests in headless or headed mode and emulate native mobile environments like Google Chrome for Android and Mobile Safari.

// playwright.config.ts import { defineConfig } from "@playwright/test"; /** * See https://playwright.dev/docs/test-configuration. */ export default defineConfig({ testDir: "./src/pages", reporter: "list", use: { baseURL: "http://localhost:5432/", }, timeout: process.env.CI ? 10000 : 4000, // ... more options });

Issues with Playwright

Fast and reliable end-to-end testing for modern web apps | Playwright

This means you cannot place test files outside of this directory, which was brought up as a question on Github some time ago. Initially, I thought it would be nice to add another folder in the repo called "scripts", but Playwright does not allow multiple testDir values.