Media-source Alternatives

media-source reviews and mentions

• Chrome 0day is being exploited now for CVE-2022-1096; update immediately
  8 projects | news.ycombinator.com | 26 Mar 2022

  It depends heavily on the website we're talking about but there's generally a lot going on when streaming video on the web.

  Usually what happens at the core is that JavaScript will download video, audio and subtitles progressively through small chunks of data called "segments" and push them to JS-exposed buffers called 'SourceBuffer'. Deciding which chunk to download, downloading them and pushing them already require a lot of JavaScript (for example, you need to decide which video and audio quality to download through adaptive algorithms, which tend to be quite complex, moreover there's also a lot of media events that needs reaction to, like when seeking, rebuffering, changing track etc.). You also have a lot of JavaScript there to limit risks of playback stalling and if you have DRMs, a lot of JavaScript there to be able to recuperate the right decryption keys (an operation you generally wish to finish as soon as possible as it is often the last step before playback).

  On some websites, you might want to play with as low latency as possible between the broadcaster and the user. In those cases, you might want to optimize your JS code, have very small checking intervals, and you might again prefer to run as much code as possible in a worker to avoid rebuffering due to the risk of the main thread being too occupied doing other things to push media segments.

  Even on non-low-latency contents, some websites which already have a lot of JavaScript running beside video playback such as at least Facebook and YouTube pushed browsers for quite some time now to be able to use the main JavaScript media streaming APIs in a worker (https://github.com/w3c/media-source/issues/175), e.g. in another thread.

  You could also have complex contents (lot of audio and subtitles languages, many audio and video qualities, multiple decryption keys, long duration etc.) that may lead to big performance and memory issue when parsing them on the JS-side. Those contents are usually described through a file named "manifest" or "playlist" which in this case can take a lot of resources to process (the document can be up to a huge 15MB XML where I work), often leading either the linked JavaScript to run in a worker or to use webassembly (a solution we chosed). Even more if you consider live contents, where this document might have to be regularly refreshed.

  You might also want to apply some processing on the media played, for example transmuxing mpeg-ts segments to MP4 ones so they can be played by more browsers. Those are very frequent operations that can be performance-sensitive and are also often performed in another thread.

  Again it very much depends on the website and I mainly know the use cases I personally encountered. Generally, adaptive media player are very complex JavaScript beasts.

  Also performance issues and poor memory management from the browser-side can lead to a lot of issues. A recurring issue at my work is bad performance leading through side-effect to a very poor quality being played (due to the high overhead in loading segments, pushing them to the buffer etc.).

  All these would suffer without a powerful and featureful JS engine like we generally have today on most browsers.