youtube-to-html5-loader VS video.js

When you embed a YouTube video on another website, it's loaded inside of an element. This iframe basically gives you a window into the code of another location; in this case, it's YouTube. It loads everything you need to watch the YouTube video inside the iframe without needing to be on YouTube.com.

That sounds great, but here's the catch: you aren't able to access or modify anything within that iframe if it's not on the same website that you're currently on. This means that if you're not on YouTube but you're watching an embedded YouTube video, you can't access the goodies inside of the iframe, such as the actual element.

TensorFlow.js needs the element to analyze the movements of the video.

Therein lies the problem: How do we get a video from YouTube into TensorFlow.js?

The Difficulties

There are two main ways to embed a YouTube video on your own website: you can either copy the embed URL (when you hit "Share" on the video) or use the Embedded Players API.

The Embedded Players API gives you a lot more control over the such as various properties (video current time, duration, paused, etc) and methods (play video, pause video, seek, etc). While this might seem great, we can't access the raw video stream contained in the element because it's in the inaccessible iframe.

The Embedded Players API would work if we didn't have to run TensorFlow.js on it, and this possibility is discussed later in the legality section.

I also looked around for other ways to extract the video stream from a YouTube video.

First was the lite-youtube-embed. This package allows you to embed YouTube videos just like normal, but they render really fast. I thought that there might be some trickery in here that I could use, but I wasn't able to find anything useful.

Next up was youtube-to-html5-loader, which sounds exactly like what we need. And unsurprisingly, it's exactly what we need!

But there's a catch: it's currently broken due to YouTube rate limiting. That's something that can be very tricky to work around, and I don't want to rely on something that could break at any point with new YouTube updates or changes.

The Solution

youtube-dl is a great way to download YouTube videos. Is there some way I can leverage this to have players download videos so that they can run the videos through TensorFlow.js and have smooth, instantaneous playback?

Yep.

There doesn't seem to be a good way for a player to simply download a YouTube video from the browser, but I have a workaround in mind. I'll have my server download the video for them and provide a link to download it.

But the storage cost could add up quick if there are a lot of videos being downloaded and stored! So a compromise here would be to store every video downloaded for a limited amount of time, say 4 hours, and then delete it. If a client needs the video again in the future, it would be re-downloaded on the server. A maximum file size cap would have to also be included in case there are a lot of videos being downloaded at once.

One more thing, though: we don't want a file download popup for users when they download the video! This will complicate things and make the video inaccessible to the browser. Instead, we can use an AJAX request to download it as a blob and store it in IndexedDB, all without a download popup! I found this post while researching that seems like a good starting place.

The end flow might look a little like this:

1. User pastes YouTube URL into the game.
2. The game server checks a few things:
   • Is this video already available for download? (if so, just serve it)
   • Is there enough available storage space to download this video? (if not, put in a waiting queue until space is available)
3. After the checks, the game server downloads the video from YouTube.
4. The game server sends the client the video link, and then the client downloads the video.
5. Now the client can instantly play back the video whenever they play or edit the song. No more waiting times! (except when loading PKD and metadata, which will be stored on the game server)

Is It Legal?

Is downloading a YouTube video onto your computer legal?

According to YouTube, it isn't. Not without permission from YouTube and the copyright owner of the video.

That's not the entire story, though. There's also a Fair Use policy that provides a gray area when dealing with these videos.

Does this browser dance game fit the Fair Use doctrine? It's difficult to answer right now.

When taking all these factors into consideration, there are a few steps that could be done that would completely remove the legal questions:

• When browsing for a song to play on the website, only stream videos from YouTube that allow embedding. Use pre-generated PKD in time with the YouTube video.
• When creating a new song through the editor, you will be required to upload a local video. The video will only be used in the browser and will not be uploaded to any external server. You may obtain this video through a variety of means and it will have to be linked to an existing YouTube video (to be streamed in the above scenario). Whether or not you download and use a YouTube video that you do not own is up to you, but it is only used in the creation purposes and won't be retained after you've created the song with its PKD.

With these potential solutions in mind, why not implement them to start?

• Editor ease-of-use: if you want to learn a cool dance you saw on YouTube, you'll first have to figure out how to download that video and then upload it. There are a lot of shady YouTube downloader sites out there.
• Gameplay smoothness and control: when playing a song, it will have to first load the embedded YouTube video. The video might play ads, which could cause issues. The YouTube Player API is also limited and might not have enough fine control over the video playback. All of the YouTube UI will also display on top of the video, which could disrupt visual feedback or cause other issues. Streaming and loading the video while playing could also cause latency and stuttering depending on the connection quality and speed.
• User base: I will probably be the only one using this, so why jump through more hoops than I have to? I'll start with the method that seems best right now, and will revisit in the future if needed.

And so while YouTube videos don't quite mix with TensorFlow.js, sometimes you just need to put them both in a blender to get a smoothie. ✨

For people who like to watch with subtitles, VLC currently doesn't support streaming to chromecast with SRT subtitles.. there are several issues for it and I believe support is slated for the next major version of Chromecast, but not sure when that will be.

The typical "workaround" is to reencode the video file to include the subtitles directly, but that sounded like too much work, so I hacked together a static page using https://videojs.com/ to embed a player and load the video and subtitles in a browser window.

Here it is in gist form if anyone has a similar issue: https://gist.github.com/HartS/9bb2721fa73b6798efcdbf5c463e87...

This was hacked together as quickly as possible for my own needs, so definitely not intended to be an example of clean code. You need to run the python server separately to serve the SRT because video-js can't load it from a file URL IIRC

Thank you! This is the kind of creative solution I was looking for. Your comment helped me find video.js which has first-party support for opening M3U8 streams.

videojs is superior to basically everything. It's also open source...

Another option is looking at https://videojs.com/ with the Vimeo video file links.

I am woring with HTML - I managed to download a (m3u8) video. by inspecting the webpage (videojs.com).

🔀 Repositório no GitHub

Now, to display the HLS stream to viewers, we’ll use HLS.js, which we installed earlier. For more UI customizations, you can check out Video.js, which uses HLS.js internally.

Maybe videojs?

You can find real-life examples of customized HTML5 video players on YouTube, the Cloudinary Video Player, JWPlayer, and Video JS. Each of these websites or frameworks utilizes the power of CSS to customize their videos or allow their users to do the same.

It looks like the IJF has switched to using the Video.js player on their portal. I'm not sure what to think of that. This could be the IJF taking more control over their IP which is ultimately a good thing. If you want to watch specific matches and see when certain actions happen then you must use their portal.