web-codecs VS meyda

Encoding alpha, please! https://github.com/w3c/webcodecs/issues/672

From developer.mozilla.org [0]

> WebCodecs API

> The WebCodecs API gives web developers low-level access to the individual frames of a video stream and chunks of audio. It is useful for web applications that require full control over the way media is processed. For example, video or audio editors, and video conferencing.

And from w3c [1]:

> The WebCodecs API allows web applications to encode and decode audio and video

All this looks really promising, I wouldn't have thought that we could use browsers directly to render videos. Maybe Puppeteer could then stream the content of the page it is rendering, for example a three.js animation.

[0] https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_A...

[1] https://github.com/w3c/webcodecs

As I understand it, Web Codecs is quite a ways away from being a web standard - it is currently just a draft[1] for a recommendation for a possible future standard.

Just beware of any major API changes or any indication that it might be dropped in its entirety!

[1]:https://w3c.github.io/webcodecs/

WebCodecs has Mozilla and Microsoft coauthors: https://w3c.github.io/webcodecs/

WebGPU was chartered in 2017, with more Apple people in the initial membership than Mozilla or Google people: https://gpuweb.github.io/admin/cg-charter.html

Google specifically has approached Apple for input about scheduler.postTask(), and they say they've presented the proposal before at a working group that Apple is part of: https://lists.webkit.org/pipermail/webkit-dev/2021-June/0319...

I won't disagree that there's a ton that can be improved about the process, but it seems pretty unsubstantiated to claim that Google is "rushing forwards as fast as possible" given that they've been spending years working on these APIs in public. You can see from the rest of the post that they're running some experiments which they're specifically choosing not to enable by default, though they could.

I work for PaperlessPost.com, and for our Flyer product we export mp4s from ffmpeg compiled to WASM, which is similar to ffmpeg.js but optimized for our use case which sounds similar. We have this WebAssembly method working well but iOS 15 (and the other major browsers) now support MediaExporter which might be a better way to go if you have something else to convert these files. This becomes and issue because you can't control the format MediaRecorder is recording to but the management of memory, the image quality, the compression, the performance hit will be more ideal than these other methods. OMGgif is very slow and will produce large files or very bad looking ones. Keep in mind that GIFs limit the colors so something that looks nice on the screen might not look the same after it is saved. The other thing to look out for is the WebCodecs APIs which should be the ultimate way to handle all of this in the future but it is only working in Chrome I think https://github.com/w3c/webcodecs

While in theory it might be possible to run ffmpeg through emscripten and run it in a web worker (I certainly assume someone has done this), it's not necessarily practical to try and use the same technique from node to transcode audio on the web. The good news is that the w3c has chartered a working group to focus on web codecs. While this is at the time of writing still in early stages, the working group is powering ahead on designing and proposing an API to enable media transcoding on the web, and hopefully that will become available to us in the near future.

Sounds like a job for web workers. Do you have any examples? I'd love to check them out. I've seen some sequencers but never a full-blown DAW attempt. I've been toying with a couple browser-based, realtime audio ML ideas lately (mostly porting some models to Tensorflow.js), so my interest is piqued.

As far as libraries go for analysis, the only solid option I've found so far is Meyda[0]. I was drawn to it mostly because it closely maps to librosa[1], and it seems fairly mature. Does anyone have any others that may come in handy for this kind of work? This is just free-time tinkering for me. I'm completely new to the space.

[0]: https://meyda.js.org/

It was pretty hacky really. There's a javascript library called Meyda that does audio processing and their front page has this visualization, i was just careful in my cropping. :). I should really give them credit in the description.

I recently published a beta build of Meyda to the npm registry, with the intention of having one of our longest running users test it out to make sure it worked in their project. I hadn't done a manual release in a long time, since we use semantic-release, so I skimmed the output of npm publish --help, and figured out what command I would run. I set the version field of package.json to 5.1.7-beta.0, as instructed built the bundle, ran our test suite, and ran npm publish . --dry-run, to verify that the manifest of files that would be published was correct. It was correct, and so I ran

If you're using Meyda to analyze audio that you load in this way, you will need to make sure that the sample rate of the audio matches the sample rate that Meyda is set to use. Otherwise you'll end up with audio features that are incorrect, and based on a skewed frequency scale. You can either match the Meyda sample rate to the wav sample rate, or you can resample the audio to fit a standard sample rate (i.e. 44,100hz, or 48,000hz). Resampling audio is a complicated topic beyond the scope of this article, but if you have trouble finding information online, let me know and I may find time to write an article.