proposals VS puppeteer

Ideally, Argon2, like PBKDF2 would just get added to SubtleCrypto (GitHub issue here), which would allow to forgo WebAssembly entirely.

I'd like to use Argon2 for web stuff but WebCrypto doesn't support it yet (https://github.com/WICG/proposals/issues/59) and the WASM flavor creates problems with bundling and testing.

ref: W3C Web Incubator Community Group (WICG)

Yeah, looking forward to switching it out if/when argon2 enters webcrypto.

> The problem is that people look at the Apple / Google duopoly and say "look we have competition! How can you compare this to Microsoft in the 90s!?"

Strong agreement. There's also a flacid, weak, shitty, useless anti-trust system, which was stripped of power by Borkism in the Reagan era. My generation has literally never seen effective anti-trust. Not once has a case been made against exploitative, anti-competitive practices that really has resonated in America, in decades, and it's not for lack of shitty, no good villians. It's because of Borkism[1][2], because of redefinition of what anti-trust meant under Reagan.

> The sheer amount of anticompetitive behavior both these companies have done over the years is insane when you put in prospective what Microsoft went to court over.

That Microsoft faced so much shit for such a relatively small act is amazing. What Apple does is absurd to me, that there's seemingly no legal challenge to their dominion-without-question of 30% of the web. I've looked at quite a number of anti-trust complaints against Google, and frankly, I owe it to myself/all to go re-review.

Google like Apple has suits against them for the 30% cut they charge at their store, which is both valid & respectable but also- on Android- easily avoidable & the open-source OS itself (& it's releases) actively supports alternative delivery platforms such as F-Droid and sideloading: Google actively supports competition. But we're seeing a lot of apps drop in-app sales, and I think that's a telling & real response: 30% is absurd. There's a number of suits against ads, and search. To be honest, none of these have left a distinct impression, have really clicked for me. I am fully able to believe there may be some serious fuckery here-abouts. There's a suit about Google Assistant systems being unable to also support alternative systems like Alexa: as a fan of general-purpose computing & competitive competition, I think this is absolutely a place that there should be straightforward & clear mandates for all companys, Google included, to be compelled to allow interopation: restricting people making devices to have to pick one and only one partner is basically a reasonable battle against Qualcomm-ism[3], against coersion, is a move to enable basic device-maker and consumer choice.

Against all of this backdrop though, one critical thing I think most of the world really has no sense of is that Google is somewhat alive under a weird patronage model. Their cash cows feel serve as patrons to the artists, and the artists are there just to make the ecosystem healthy & alive. The cash registers ring because of a semi-open market, because the web is a pretty damned good place to connect, host shit, do shit; better than AOL, better than Microsoft Windows, better than apps.

Trying to stack the deck laterally, to make the web be Google's web, or Android be Google's Android: they are extremely liable to kill the cash cow. These need to be healthy, independent, functional systems, that are getting better & remaining at the very forefront of competition against all others. This health is absolutely the pinnacle concern, is existentially important. Android or the web could readily collapse if things go poorly, if corruption takes root, if whiffs of real genuine misdealings gets into the air. And frankly, the problem solves itself internally. Google historically & famously has been an engineering lead organization. They have a long history of employing very good, public figures who care about the web, who know about the web, who have wanted to make the web better, who seem motivated by strong intrinsic desires. These people sit on standards boards, they help align Chrome. These people don't take shit from traditional corporate lackies trying to make some fast bucks by dodgy inter-dealings.

Again, given Google's strong first & second party relationships (search and ads), no matter what happens with browsers, webtech (& to a lesser degree Android), Google will have an incomparable vast & mighty perch to understand & analyze & model the web from, and not unfairly, not by cheating, not by underhandedness. The objective is to keep the shared, common, competitive platform alive, shared, & competitive. By doing good engineering. By making development better & easier & giving them better superpowers. By not hazarding gross breakage that would sabotage public belief/faith. This isn't a super complex system. It's nicely isolated parts, which each do their own thing: make better systems, use free-market search & ads to be top of the game (to make $ & to keep funding/patronizing the essential ecosystem).

> Many of us saw projects like Firefox grow in the 2000s, giving hope that open source and standards would win out in the end. But we dwindled, lulled by the sweet promises of Chrome and their open core.

True true. Will has been lost. Chromium is remarkably accessible, there are remarkably good hooks still in place to go build our own sync systems or what not. Few have chased up, have tried to really amplify & enhance Chromium into an open browser. That's unfortunate.

In general I'd say the real frontier for advancement is on https://wicg.io . This has been a very compelling case for how the web really needed to advance all along. An extremely low barrier to entry to start proposing ideas, where other standards folks & standards-adjacent folk can chime in & help steer, help sheppard young web ideas into desireable, promising standards that stand a fair chance of being adopted.

I definitely wish there were more alternatives out there, more efforts. I have hope we'll see some new arisals show up. But at the same time, I don't see Chrome as bad or scary or problematic. There's very few cases people have made against it that seem, well, real. Emotion & fear & doubt run rampant. Even when the team makes decisions I truly detest (e.g. squandering awesome HTTP Push potential then abandoning the capability) I generally understand & can see where folks are coming from. We're not accelerating to where I'd wish to go, but it's incredibly rare that I see Chrome/Chromium as going in actively bad directions, building "bad" web platform. Very few have made a case that I can really see or grasp, that's worth agreeing or disagreeing with about Chrome or Android, about how Google invests & shapes these forwards. I continue to see this more as a patronage system, as investment in the necessary & worthy ecosystem, that supports the existence of a separate, more corporate entity. And I don't see the anti-competitive practices taking root in this web or android space, generally.

[1] https://pluralistic.net/2021/02/06/calera/#fuck-bork

[2] https://pluralistic.net/2022/05/09/rest-in-piss-robert-bork/...

[3] https://news.ycombinator.com/item?id=31628094

> I'm so tired of Google inventing arbitrary stuff for the web, implementing it, and everyone then expecting Mozilla and Apple to just follow suit.

I agree, the situation is awful.

Personally, I'm so tired of Apple & Mozilla having complete veto power on the web.

The lack of will to standardize, the push back against doing good things, is way way way way way too high. Apple has long been the foot dragging, complaining, miserable laggard, with lots of their own ideas & little willingness to play with others. Mozilla recently seems to have switched sides: having given up on WebOS, they now seem to spend more time claiming to be "pro-privacy" & fighting against really good new capabilities than working things forwards. Apple and Mozilla both know Google is widely revilved & mistrusted, and are marketing the shit out of their anti-platform stances.

Thankfully we see a ton of really really good work happening on https://wicg.io. It's a good neutral territory to make the platform better. Tons of the web improvements happening are vendor-neutral, are just people with good ideas, that eventually, with support & community, stand a chance of getting sensible improvements implemented. Alas, Apple. :'(

> This means that software development is a failing discipline. All we can do is to come up with ever more ridiculous layers of complexity on top of mindboggling complexity, recursively. All of this is totally self-inflicted. The problem domains that our software needs to deal with are complex enough. But we keep piling on shit of our own making.

Awwhhh why so glum? This sounds so alarmist, so fraught & bleak!

I don't see anyone as having a bead yet on what the final destination is, on what is right or perfect. I see change & innovation & exploration as necessary, ongoing, and this layering of platforms atop each other is part of that larger bigger quest for us all to learn what serves ourselves well, to figure out how we align.

Overall these Adapter layers are quite performant, quite fast, and they isolate rather than leak complexity quite effectively. Electron's doesn't have to invent a ton of stuff to create this pleasant, familiar environment for developers or users- the operating system is simply not that relevant, is easily adapted, by a pretty boring regular programming language (Node.JS) and the world's most popular multimedia page/resource system (the browser).

Right now, yes, we have the web platform as a layer above native in many cases, but I'm not sure that that is so alarming. Maybe it's transitional? The early smartphones were both enormously web first, decided to use something great rather than reinvent: the Palm Pre and original iPhone (which was webapps only). In modern times, there's ChromeOS, and Palm's webOS continues under new stewartship, mainly on TVs. If the complaint is layers of complexity, maybe we just need to get rid of proprietary & legacy platforms, & embrace the common, shared medium that all computing has. Or find new unifying better platforms!

In general, I see these new platforms as being extremely liberating, as helping reduce the complexity developers have to mess with, by offering a set of well-defined standards & well known constraints & behaviors. Rather than a complicated, particular OS tied to some specific devices, with it's own quirks, with ever evolving platform capabilities & changing toolchain

> If this is what software development has become, then there's no way it won't end in total and utter disaster when people start to recognize the already all too common piss poor products that barely manage to do the absolute minimum and rebel against them. It may not be marketing department of the Sirius Cybernetic Corporation who'll be lined up against the wall first, but software developers when the revolution comes.

Again I think moderation is good council here, but I also agree in principle that there is some awareness software can be a bit of a disaster, and it's visible that sometimes updates & "improvements" serve external interests not the users.

Where I differ is I see Electron as a fairly hopeful, possible open future for computing, that does embrace users, more so than most software. Most software is not malleable, not adaptable. Electron, on the other hand, offers the very slick, open ended DevTools Protocol for most every app, which allows users control & automation & expansion of software. We can write some userscript & change the behavior of webplatform & electron systems, which is hugely powerful, is a far fairer shake & far more liberty than most computing platforms, where apps are usually compiled down, fixed in form & nature.

This second paragraph really brings me back to where we started: I don't think we have a bead yet, as a larger world. It feels like there's so much discovery, so much understanding to develop. What makes me hopeful is groups like https://wicg.io , which try to understand & consider how we might do better, which work to build open standards for the internet, for our shared multimedia platform, as a community, securely. There's so much further for all computing to go, so much we need to do to better serve users. I don't see anyone has having a strong lock on that, it feels like there's more to learn & become than there is that we've locked down, by far, so I am hopeful & excited & happy to see us staying malleable, working to unblock innovation at all levels, in all computing. Electron is a great & positive force here, albiet I look forward to other projects listed in the comments such as Tau providing similar-ideas with slightly-different execution.

The problem is certificates and WAN access, and lack of MDNS on Android. There's basically no way to do anything that doesn't involve some manual setup, aside from developing a new purpose built app, and maintaining it in addition to the product, probably on two platforms.

If Mozilla still had FlyWeb things could be plug and play.

I have a set of proposals here to bring some of that back: https://github.com/WICG/proposals/issues/43

And some are considering the Tox protocol, but in general, we have not solved the most basic issue of self hosting. How do I connect to my device in a way that just works, LAN or WAN, without manually serting up the client or registering for a service?

Yes, it is 100% time fot that.

Mozilla FlyWeb was close, but didn't handle remote access. It could have been extended to, but they dropped it.

I wrote a proposal here for how this could be done with Bluetooth-like pairing, using URLs that embed a certificate hash, a random sequence as an extra security layer, and a lookup URL that one can ask where to find the host for a service, for access over the WAN.

By using a URL instead of normal non-HTTP DNS, the lookup URL can be another web service self hosted using the same TLD, or a Data URI if one is hosting from a static IP.

No part of the URL besides the key is used to determine the origin for CORS and local storage, so you can change discovery methods and the random string freely.

Initial connection is by directly sending a link, or by LAN discovery.

Unless discovery is enabled and you are on the same network, it should be impossible to connect without already knowing the URL, so even if your home automation hub is very badly coded, they can't even start hacking it till they find your URL, which can't be found just by sniffing(Because of that random string).

Clients track the "last seen" address of servers, so even if lookup goes down, access still works until your home IP changes.

When nodes connect over LAN, the server sends it's "Find me on the WAN at" IP. So even with no discovery server at all and no static IP, it creates a very convincing illusion of "just working" 99% of the time.

Which means that if you buy a device that uses a cloud lookup service, and they drop that service, your device will still be remote accessible, most of the time. Which might be good enough, or at least good enough to get by until you can find a more permanent solution.

Proposal:

https://github.com/WICG/proposals/issues/43

And a partial implementation of a very close version(Lookups always use OpenDHT in this), plus a notetaking app based on it.

https://github.com/EternityForest/hardlinep2p

I really think this is one of those critical missing technologies that would really enable a lot of amazing things.

1.Puppeteer: Puppeteer is a Node library that provides a high-level API to control headless Chrome or Chromium using the DevTools Protocol. Key Features: More control over Chrome. Enables web scraping. Allows taking screenshots and generating PDFs for UI testing. Measures load times through the Chrome Performance Analysis tool

HTML to PDF conversion is a common requirement in modern web applications. It allows users to save web pages, reports, and other content in a format that is easy to share and print. There are many libraries and services available for converting HTML to PDF, each with its own strengths and weaknesses. In this article, we will compare some of the most popular HTML to PDF renderers in Node.js, including Puppeteer, Playwright, node-html-pdf, and Onedoc.

Playwright seems to be a superior library for working with headless browsers than Puppeteer, but I will go with Puppeteer.

Puppeteer is a Node library that provides a high-level API to control headless Chrome or Chromium. It's primarily used for browser automation, making it a powerful tool for end-to-end testing of web applications, taking screenshots, and generating pre-rendered content from web pages.

Puppeteer

Website: pptr.dev Repositório: GitHub

# Define variables PYTHON := python3 POETRY := poetry PYTEST := pytest PIP := pip3 PROJECT_NAME := web automation with Pyppeteer .PHONY: install install: $(POETRY) install @echo "Dependency installation complete" $(PIP) install -r requirements.txt @echo "Set env vars LT_USERNAME & LT_ACCESS_KEY" # Procure Username and AccessKey from https://accounts.lambdatest.com/security export LT_USERNAME=himansh export LT_ACCESS_KEY=Ia1MiqNfci .PHONY: install poetry-install: poetry install .PHONY: test test: export NODE_ENV = test .PHONY: test pyunit-pyppeteer: - echo $(EXEC_PLATFORM) - $(PYTHON) tests/pyunit-pyppeteer/test_pyunit_pyppeteer.py .PHONY: test pytest-pyppeteer: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s -n 2 tests/pytest-pyppeteer/test_pytest_pyppeteer_1.py \ tests/pytest-pyppeteer/test_pytest_pyppeteer_2.py .PHONY: test pyunit-pyppeteer-browser-session: - echo $(EXEC_PLATFORM) - $(PYTHON) tests/starting-browser-session/pyunit/test_pyppeteer_browser_session.py .PHONY: test pytest-pyppeteer-browser-session: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s \ tests/starting-browser-session/pytest/test_pyppeteer_browser_session.py .PHONY: test asyncio-run-pyppeteer-browser-session: - echo $(EXEC_PLATFORM) - $(PYTHON) tests/starting-browser-session/asyncio_run/test_pyppeteer_browser_session.py .PHONY: test asyncio-run-complete-pyppeteer-browser-session: - echo $(EXEC_PLATFORM) - $(PYTHON) tests/starting-browser-session/\ asyncio_run_until_complete/test_pyppeteer_browser_session.py .PHONY: test pyppeteer-button-click: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/button-click/test_page_class_click.py .PHONY: test pyppeteer-activate-tab: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/active-tab/test_page_class_bringtofront.py ###### Testing Custom Environment - https://miyakogi.github.io/pyppeteer/reference.html#environment-variables # Available versions: 113, 121, and default .PHONY: test pyppeteer-custom-chromium-version: - echo $(EXEC_PLATFORM) - echo 'Browser Version:' $(CHROMIUM_VERSION) - $(PYTEST) --verbose --capture=no -s tests/custom-configuration/test_launcher_exe_path.py ###### Testing Headless - https://miyakogi.github.io/pyppeteer/reference.html#launcher # Available values: headless and non-headless .PHONY: test pyppeteer-custom-browser-mode: - echo $(EXEC_PLATFORM) - echo $(BROWSER_MODE) - $(PYTEST) --verbose --capture=no -s tests/custom-configuration/test_launcher_headless.py .PHONY: test pyppeteer-generate-pdf: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/generate-pdf/test_page_class_pdf.py .PHONY: test pyppeteer-generate-screenshot: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/generate-screenshots/test_page_class_screenshot.py .PHONY: test pyppeteer-cookies: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/handling-cookies/test_page_class_cookies.py .PHONY: test pyppeteer-dialog-box: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/handling-dialog-box/test_handling_dialog_box.py .PHONY: test pyppeteer-iframe: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/handling-iframe/test_page_class_iframe.py # Like Puppeteer, Navigation operations mentioned below only work in Headless mode # goBack: https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goBack # goForward: https://miyakogi.github.io/pyppeteer/reference.html#pyppeteer.page.Page.goForward # Bug Link # https://github.com/puppeteer/puppeteer/issues/7739 # https://stackoverflow.com/questions/65540674/how-to-error-check-pyppeteer-page-goback .PHONY: test pyppeteer-navigate-ops: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/navigate-operations/test_page_class_navigation_ops.py .PHONY: test pyppeteer-request-response: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/request-response/test_page_class_req_resp.py .PHONY: test pyppeteer-viewport: - echo $(EXEC_PLATFORM) - echo $(BROWSER_MODE) - $(PYTEST) --verbose --capture=no -s tests/setting-useragent-viewports/\ test_page_class_useragent_viewport.py::test_mod_viewport .PHONY: test pyppeteer-non-headless-useragent: - echo $(EXEC_PLATFORM) - echo $(BROWSER_MODE) - $(PYTEST) --verbose --capture=no -s tests/setting-useragent-viewports/\ test_page_class_useragent_viewport.py::test_get_nonheadless_user_agent .PHONY: test pyppeteer-headless-useragent: - echo $(EXEC_PLATFORM) - echo $(BROWSER_MODE) - $(PYTEST) --verbose --capture=no -s tests/setting-useragent-viewports/\ test_page_class_useragent_viewport.py::test_get_headless_user_agent .PHONY: test pyppeteer-dynamic-content: - echo $(EXEC_PLATFORM) - echo $(BROWSER_MODE) - $(PYTEST) --verbose --capture=no -s -n 4 tests/handling-dynamic-content/\ test_page_class_lazy_loaded_content.py .PHONY: test pyppeteer-web-scraping: - echo $(EXEC_PLATFORM) - $(PYTEST) --verbose --capture=no -s tests/web-scraping-content/\ test_scraping_with_pyppeteer.py .PHONY: clean clean: # This helped: https://gist.github.com/hbsdev/a17deea814bc10197285 find . | grep -E "(__pycache__|\.pyc$$)" | xargs rm -rf rm -rf .pytest_cache/ @echo "Clean Succeeded" .PHONY: distclean distclean: clean rm -rf venv .PHONY: help help: @echo "" @echo "install : Install project dependencies" @echo "clean : Clean up temp files" @echo "pyunit-pyppeteer : Running Pyppeteer tests with Pyunit framework" @echo "pytest-pyppeteer : Running Pyppeteer tests with Pytest framework" @echo "pyunit-pyppeteer-browser-session : Browser session using Pyppeteer and Pyunit" @echo "pytest-pyppeteer-browser-session : Browser session using Pyppeteer and Pytest" @echo "asyncio-run-pyppeteer-browser-session : Browser session using Pyppeteer (Approach 1)" @echo "asyncio-run-complete-pyppeteer-browser-session : Browser session using Pyppeteer (Approach 2)" @echo "pyppeteer-button-click : Button click demo using Pyppeteer" @echo "pyppeteer-activate-tab : Switching browser tabs using Pyppeteer" @echo "pyppeteer-custom-chromium-version : Custom Chromium version with Pyppeteer" @echo "pyppeteer-custom-browser-mode : Headless and non-headless test execution with Pyppeteer" @echo "pyppeteer-generate-pdf : Generating pdf using Pyppeteer" @echo "pyppeteer-generate-screenshot : Generating page & element screenshots with Pyppeteer" @echo "pyppeteer-cookies : Customizing cookies with Pyppeteer" @echo "pyppeteer-dialog-box : Handling Dialog boxes with Pyppeteer" @echo "pyppeteer-iframe : Handling iFrames with Pyppeteer" @echo "pyppeteer-navigate-ops : Back & Forward browser operations with Pyppeteer" @echo "pyppeteer-request-response : Request and Response demonstration using Pyppeteer" @echo "pyppeteer-viewport : Customizing viewports using Pyppeteer" @echo "pyppeteer-non-headless-useragent : Customizing user-agent (with browser in headed mode) using Pyppeteer" @echo "pyppeteer-headless-useragent : Customizing user-agent (with browser in headless mode) using Pyppeteer" @echo "pyppeteer-dynamic-content : Handling dynamic web content using Pyppeteer" @echo "pyppeteer-web-scraping : Dynamic web scraping using Pyppeteer"

This library works by creating an instance of WhatsApp web running inside an instance of headless chrome automated by puppeteer. In my testing, I ran into tons of compatibility issues when trying to use these dependencies inside anything other than a bare-bones Node.js + express server. Also, we can’t spin up a new instance of chrome and WhatsApp web each time a user sends a message, this will exhaust our allowed WhatsApp connections (4 max), not to mention that doing this will make the response times painfully slow.

It's a tricky balance to strike. Obviously, krono, you're right: ff we're being generous the app launch GA tracker they use, they probably consider it shares nothing of value or identifiable, but strictly speaking it's not true that no data shared.

I think the audit you do is actually valuable. In my company's product, BrowserBox^0 (also a browser, funnily enough! focused on remote isolation, privacy & security, and including Tor support), we don't collect anything but we want to add some kind of telemetry regarding usage so we can even basically know how many daily users we have (outside of licensed channels where it's tracked in the contract).

Even tho we don't collect anything, we don't advertise "Zero data collection" anywhere, because of how sensitive I think this topic is. I think we really need to be solid on it, if we're going to say that. And to cover us, in the privacy policy, we say "we may collect some data for operational purposes to ensure the continued running of the service" (paraphrasing), even tho we don't.

One niggle is that we use Chrome in headless mode (can also use Edge / Chromium), and while we appreciate the auto updating quality that Chrome has for security patches, we're cautious that maybe Chrome still collects telemetry and sends it to Google even with headless and correct flags (see for instance: https://github.com/puppeteer/puppeteer/blob/afb7d9eb5854e851...)

I know it's not your job...but what's your advice on how to proceed specifically?

0: https://github.com/BrowserBox/BrowserBox

If you've read my previous blog posts or ever experimented with Secutils.dev, you might be familiar with the web resources tracking utility. This utility allows you to monitor changes in web page resources, specifically JavaScript and CSS. While it has a somewhat narrow security-focused purpose — detecting broken or tampered web application deployments — it may not be the type of tool you use daily. Nevertheless, it serves as a good example of what you can build with modern browser automation tools like Playwright and Puppeteer. If you're interested in digging deeper into this specific utility, refer to the following blog post series: