YouTube Videos Don't Mix With TensorFlow.js

• youtube-to-html5-loader

  1 132 0.0 JavaScript

  Load YouTube videos with the HTLML5 <video> element without needing iframes or the YouTube JS API.

When you embed a YouTube video on another website, it's loaded inside of an element. This iframe basically gives you a window into the code of another location; in this case, it's YouTube. It loads everything you need to watch the YouTube video inside the iframe without needing to be on YouTube.com.

That sounds great, but here's the catch: you aren't able to access or modify anything within that iframe if it's not on the same website that you're currently on. This means that if you're not on YouTube but you're watching an embedded YouTube video, you can't access the goodies inside of the iframe, such as the actual element.

TensorFlow.js needs the element to analyze the movements of the video.

Therein lies the problem: How do we get a video from YouTube into TensorFlow.js?

The Difficulties

There are two main ways to embed a YouTube video on your own website: you can either copy the embed URL (when you hit "Share" on the video) or use the Embedded Players API.

The Embedded Players API gives you a lot more control over the such as various properties (video current time, duration, paused, etc) and methods (play video, pause video, seek, etc). While this might seem great, we can't access the raw video stream contained in the element because it's in the inaccessible iframe.

The Embedded Players API would work if we didn't have to run TensorFlow.js on it, and this possibility is discussed later in the legality section.

I also looked around for other ways to extract the video stream from a YouTube video.

First was the lite-youtube-embed. This package allows you to embed YouTube videos just like normal, but they render really fast. I thought that there might be some trickery in here that I could use, but I wasn't able to find anything useful.

Next up was youtube-to-html5-loader, which sounds exactly like what we need. And unsurprisingly, it's exactly what we need!

But there's a catch: it's currently broken due to YouTube rate limiting. That's something that can be very tricky to work around, and I don't want to rely on something that could break at any point with new YouTube updates or changes.

The Solution

youtube-dl is a great way to download YouTube videos. Is there some way I can leverage this to have players download videos so that they can run the videos through TensorFlow.js and have smooth, instantaneous playback?

Yep.

There doesn't seem to be a good way for a player to simply download a YouTube video from the browser, but I have a workaround in mind. I'll have my server download the video for them and provide a link to download it.

But the storage cost could add up quick if there are a lot of videos being downloaded and stored! So a compromise here would be to store every video downloaded for a limited amount of time, say 4 hours, and then delete it. If a client needs the video again in the future, it would be re-downloaded on the server. A maximum file size cap would have to also be included in case there are a lot of videos being downloaded at once.

One more thing, though: we don't want a file download popup for users when they download the video! This will complicate things and make the video inaccessible to the browser. Instead, we can use an AJAX request to download it as a blob and store it in IndexedDB, all without a download popup! I found this post while researching that seems like a good starting place.

The end flow might look a little like this:

1. User pastes YouTube URL into the game.
2. The game server checks a few things:
   • Is this video already available for download? (if so, just serve it)
   • Is there enough available storage space to download this video? (if not, put in a waiting queue until space is available)
3. After the checks, the game server downloads the video from YouTube.
4. The game server sends the client the video link, and then the client downloads the video.
5. Now the client can instantly play back the video whenever they play or edit the song. No more waiting times! (except when loading PKD and metadata, which will be stored on the game server)

Is It Legal?

Is downloading a YouTube video onto your computer legal?

According to YouTube, it isn't. Not without permission from YouTube and the copyright owner of the video.

That's not the entire story, though. There's also a Fair Use policy that provides a gray area when dealing with these videos.

Does this browser dance game fit the Fair Use doctrine? It's difficult to answer right now.

When taking all these factors into consideration, there are a few steps that could be done that would completely remove the legal questions:

• When browsing for a song to play on the website, only stream videos from YouTube that allow embedding. Use pre-generated PKD in time with the YouTube video.
• When creating a new song through the editor, you will be required to upload a local video. The video will only be used in the browser and will not be uploaded to any external server. You may obtain this video through a variety of means and it will have to be linked to an existing YouTube video (to be streamed in the above scenario). Whether or not you download and use a YouTube video that you do not own is up to you, but it is only used in the creation purposes and won't be retained after you've created the song with its PKD.

With these potential solutions in mind, why not implement them to start?

• Editor ease-of-use: if you want to learn a cool dance you saw on YouTube, you'll first have to figure out how to download that video and then upload it. There are a lot of shady YouTube downloader sites out there.
• Gameplay smoothness and control: when playing a song, it will have to first load the embedded YouTube video. The video might play ads, which could cause issues. The YouTube Player API is also limited and might not have enough fine control over the video playback. All of the YouTube UI will also display on top of the video, which could disrupt visual feedback or cause other issues. Streaming and loading the video while playing could also cause latency and stuttering depending on the connection quality and speed.
• User base: I will probably be the only one using this, so why jump through more hoops than I have to? I'll start with the method that seems best right now, and will revisit in the future if needed.

And so while YouTube videos don't quite mix with TensorFlow.js, sometimes you just need to put them both in a blender to get a smoothie. ✨

• youtube-dl

  1,082 128,720 8.5 Python

  Command-line program to download videos from YouTube.com and other video sites

  

When you embed a YouTube video on another website, it's loaded inside of an element. This iframe basically gives you a window into the code of another location; in this case, it's YouTube. It loads everything you need to watch the YouTube video inside the iframe without needing to be on YouTube.com.

That sounds great, but here's the catch: you aren't able to access or modify anything within that iframe if it's not on the same website that you're currently on. This means that if you're not on YouTube but you're watching an embedded YouTube video, you can't access the goodies inside of the iframe, such as the actual element.

TensorFlow.js needs the element to analyze the movements of the video.

Therein lies the problem: How do we get a video from YouTube into TensorFlow.js?

The Difficulties

There are two main ways to embed a YouTube video on your own website: you can either copy the embed URL (when you hit "Share" on the video) or use the Embedded Players API.

The Embedded Players API gives you a lot more control over the such as various properties (video current time, duration, paused, etc) and methods (play video, pause video, seek, etc). While this might seem great, we can't access the raw video stream contained in the element because it's in the inaccessible iframe.

The Embedded Players API would work if we didn't have to run TensorFlow.js on it, and this possibility is discussed later in the legality section.

I also looked around for other ways to extract the video stream from a YouTube video.

First was the lite-youtube-embed. This package allows you to embed YouTube videos just like normal, but they render really fast. I thought that there might be some trickery in here that I could use, but I wasn't able to find anything useful.

Next up was youtube-to-html5-loader, which sounds exactly like what we need. And unsurprisingly, it's exactly what we need!

But there's a catch: it's currently broken due to YouTube rate limiting. That's something that can be very tricky to work around, and I don't want to rely on something that could break at any point with new YouTube updates or changes.

The Solution

youtube-dl is a great way to download YouTube videos. Is there some way I can leverage this to have players download videos so that they can run the videos through TensorFlow.js and have smooth, instantaneous playback?

Yep.

There doesn't seem to be a good way for a player to simply download a YouTube video from the browser, but I have a workaround in mind. I'll have my server download the video for them and provide a link to download it.

But the storage cost could add up quick if there are a lot of videos being downloaded and stored! So a compromise here would be to store every video downloaded for a limited amount of time, say 4 hours, and then delete it. If a client needs the video again in the future, it would be re-downloaded on the server. A maximum file size cap would have to also be included in case there are a lot of videos being downloaded at once.

One more thing, though: we don't want a file download popup for users when they download the video! This will complicate things and make the video inaccessible to the browser. Instead, we can use an AJAX request to download it as a blob and store it in IndexedDB, all without a download popup! I found this post while researching that seems like a good starting place.

The end flow might look a little like this:

1. User pastes YouTube URL into the game.
2. The game server checks a few things:
   • Is this video already available for download? (if so, just serve it)
   • Is there enough available storage space to download this video? (if not, put in a waiting queue until space is available)
3. After the checks, the game server downloads the video from YouTube.
4. The game server sends the client the video link, and then the client downloads the video.
5. Now the client can instantly play back the video whenever they play or edit the song. No more waiting times! (except when loading PKD and metadata, which will be stored on the game server)

Is It Legal?

Is downloading a YouTube video onto your computer legal?

According to YouTube, it isn't. Not without permission from YouTube and the copyright owner of the video.

That's not the entire story, though. There's also a Fair Use policy that provides a gray area when dealing with these videos.

Does this browser dance game fit the Fair Use doctrine? It's difficult to answer right now.

When taking all these factors into consideration, there are a few steps that could be done that would completely remove the legal questions:

• When browsing for a song to play on the website, only stream videos from YouTube that allow embedding. Use pre-generated PKD in time with the YouTube video.
• When creating a new song through the editor, you will be required to upload a local video. The video will only be used in the browser and will not be uploaded to any external server. You may obtain this video through a variety of means and it will have to be linked to an existing YouTube video (to be streamed in the above scenario). Whether or not you download and use a YouTube video that you do not own is up to you, but it is only used in the creation purposes and won't be retained after you've created the song with its PKD.

With these potential solutions in mind, why not implement them to start?

• Editor ease-of-use: if you want to learn a cool dance you saw on YouTube, you'll first have to figure out how to download that video and then upload it. There are a lot of shady YouTube downloader sites out there.
• Gameplay smoothness and control: when playing a song, it will have to first load the embedded YouTube video. The video might play ads, which could cause issues. The YouTube Player API is also limited and might not have enough fine control over the video playback. All of the YouTube UI will also display on top of the video, which could disrupt visual feedback or cause other issues. Streaming and loading the video while playing could also cause latency and stuttering depending on the connection quality and speed.
• User base: I will probably be the only one using this, so why jump through more hoops than I have to? I'll start with the method that seems best right now, and will revisit in the future if needed.

And so while YouTube videos don't quite mix with TensorFlow.js, sometimes you just need to put them both in a blender to get a smoothie. ✨

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project