web-codecs VS localtls

Encoding alpha, please! https://github.com/w3c/webcodecs/issues/672

From developer.mozilla.org [0]

> WebCodecs API

> The WebCodecs API gives web developers low-level access to the individual frames of a video stream and chunks of audio. It is useful for web applications that require full control over the way media is processed. For example, video or audio editors, and video conferencing.

And from w3c [1]:

> The WebCodecs API allows web applications to encode and decode audio and video

All this looks really promising, I wouldn't have thought that we could use browsers directly to render videos. Maybe Puppeteer could then stream the content of the page it is rendering, for example a three.js animation.

[0] https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_A...

[1] https://github.com/w3c/webcodecs

As I understand it, Web Codecs is quite a ways away from being a web standard - it is currently just a draft[1] for a recommendation for a possible future standard.

Just beware of any major API changes or any indication that it might be dropped in its entirety!

[1]:https://w3c.github.io/webcodecs/

WebCodecs has Mozilla and Microsoft coauthors: https://w3c.github.io/webcodecs/

WebGPU was chartered in 2017, with more Apple people in the initial membership than Mozilla or Google people: https://gpuweb.github.io/admin/cg-charter.html

Google specifically has approached Apple for input about scheduler.postTask(), and they say they've presented the proposal before at a working group that Apple is part of: https://lists.webkit.org/pipermail/webkit-dev/2021-June/0319...

I won't disagree that there's a ton that can be improved about the process, but it seems pretty unsubstantiated to claim that Google is "rushing forwards as fast as possible" given that they've been spending years working on these APIs in public. You can see from the rest of the post that they're running some experiments which they're specifically choosing not to enable by default, though they could.

I work for PaperlessPost.com, and for our Flyer product we export mp4s from ffmpeg compiled to WASM, which is similar to ffmpeg.js but optimized for our use case which sounds similar. We have this WebAssembly method working well but iOS 15 (and the other major browsers) now support MediaExporter which might be a better way to go if you have something else to convert these files. This becomes and issue because you can't control the format MediaRecorder is recording to but the management of memory, the image quality, the compression, the performance hit will be more ideal than these other methods. OMGgif is very slow and will produce large files or very bad looking ones. Keep in mind that GIFs limit the colors so something that looks nice on the screen might not look the same after it is saved. The other thing to look out for is the WebCodecs APIs which should be the ultimate way to handle all of this in the future but it is only working in Chrome I think https://github.com/w3c/webcodecs

While in theory it might be possible to run ffmpeg through emscripten and run it in a web worker (I certainly assume someone has done this), it's not necessarily practical to try and use the same technique from node to transcode audio on the web. The good news is that the w3c has chartered a working group to focus on web codecs. While this is at the time of writing still in early stages, the working group is powering ahead on designing and proposing an API to enable media transcoding on the web, and hopefully that will become available to us in the near future.

Here we are again, locking useful features behind HTTPS :)

Background: I have been working on a software that creates and streams extra virtual displays via the local network (like Duet Display or Apple's sidecar), and the easiest way to get started for everyone is to stream to a web browser.

WebRTC is kinda easy to implement thanks to webrtc-rs, but has unacceptable latency as I do not have control over whether or when exactly a frame is rendered (I need to be able to drop outdated frames). This API has the potential to do exactly what I want, but:

* It is not possible to connect to a local server if the viewer page is served via HTTPS, as the local server won't have HTTPS.

* It is possible to generate CAs and certificates locally and instruct users to install them, but I don't think that anyone out there will like this solution.

* We have WebTransport that might be able to do this, but it is an overly complex technology that uses HTTP/3 (plus no support on anything but Chromium), and tools like localtls[1] that requires internet connection and my own domain (I would really want this tool to be able to run completely offline).

Modern web APIs are kinda useless when you don't have internet, but the ship has sailed.

[1] https://github.com/Corollarium/localtls