automatic-video-processing VS natural-language-youtube-search

I'm one of the creators of Sieve, and I'm looking for general thoughts on this problem.

https://sievedata.com seems very promising, a search engine for videos, with specific tags, sounds like a very good idea.

I'd like the same for all my photos and videos: that would be so much easier to find specific pictures by keywords

The reason I ask is because I'm working on something called Sieve. We're basically making it really easy for any software developer to process and understand video content. This includes applications from home security, to pet monitoring, baby monitoring, sports analytics, and media understanding.

Hey everyone! I'm the creator of Sieve, an API for AI-based video understanding. One application we're starting to support in beta is tracking player / object movements, speed, etc in video games. All you do is push video to our API, which we then process, after which you can search + query using API calls. We're starting by supporting a few popular games like League of Legends, Dota 2, CSGO, and Overwatch. Here are the docs.

For context I'm the founder of a company called Sieve which is starting to work with some of these tools to automatically gauge things like attentiveness and facial expressions by automatically analyzing the video. Would be interesting to hear what you as users actually want.

P.S. I am potentially very bias because I'm working on Sieve which is trying to work with these applications.

Hey everyone! I recently started building Sieve, a really easy way for devs to understand video content. We've just started to work with quite a few video editing tools / companies (both online and offline ones) after having primarily focused on real-world applications like security, supply chain, and general media.

Hey everyone! I’m one of the creators of Sieve. I posted about it here a while back and thought I'd share that r/MachineLearning can now try it for free :)

Hi HN, we’re Mokshith and Abhi from Sieve (https://sievedata.com). We’re building an API that lets you add video search to internal tools or customer applications, instantly. Sieve can process 24 hours of video in less than 10 minutes, and makes it easy to search video by detected objects / characteristics, motion data, and visual similarity. You can use our models out of the box, or plug-in your own model endpoints into our infrastructure. Models can mean any software that produces output given an image.

Every industry from security, to media, supply chain, construction, retail, sports, and agriculture is being transformed by video analytics—but setting up the infrastructure to process video data quickly is difficult. Having to deal with video ingestion pipelines, computer-vision model training, and search functionality is not pretty. We’re building a platform that takes care of all of this so teams can focus on their domain-expertise, building industry-specific software.

We met in high school, and were on the robotics team together. It was our first exposure to computer vision, and something we both deeply enjoyed. We ended up going to UC Berkeley together and worked on computer vision at places like Scale AI, Niantic, Ford, NVIDIA, Microsoft, and Second Spectrum. We were initially trying to solve problems for ourselves as computer vision developers but quickly realized the unique problems in video having to do with cost, efficiency, and scale. We also realized how important video would be in lots of verticals, and saw an opportunity to build infrastructure which wouldn’t have to be rebuilt by a fullstack dev at any company again.

Let’s take the example of cloud software for construction which might include tons of features from asset trackers to rental management and compliance checks. It doesn’t make sense for them to build their own video processing for telematics—the density and scale of video make this a difficult task. A single 30 FPS camera generates over 2.5M frames within a day of recording. Imagine this across thousands of cameras and many weeks of footage—not to mention the actual vertical-specific software they’re building for end users.

Sieve takes care of everything hard about processing and searching video. Our API allows you to process and search video with just two API calls. We use filtering, parallelization, and interpolation techniques to keep costs low, while being able to process 24 hours of video in under 10 minutes. Users can choose from our pre-existing set of models, or use their own models with our video processing engine. Our pricing can range anywhere from $0.08-$0.45 per minute of video processed based on the models clients are interested in and usage volume. Our FAQ page (https://sievedata.com/faq) explains these factors in more detail.

Our backend is built on serverless functions. We split each video into individual chunks which are processed in parallel and passed through multiple layers of filters to determine which chunks are “important”. We’re able to algorithmically ignore parts of video which are static, or change minimally, and focus on the parts that contain real action. We then run more expensive models on the most “important” parts of video, and interpolate results across frames to return information to customers at 30 FPS granularity. Our customers simply push signed video URLs to our platform, and this happens automatically. You can then use our API to query for intervals of interest.

We haven’t built an automated sign up flow yet because of our focus on the core product, but we still wanted to give all of you the chance to try Sieve on your own videos for free. You’ll be emailed a personal, limited-access API key.

Try it out: https://sieve-data.notion.site/Trying-Sieve-s-Video-Search-4...

Visual dashboard demo: https://www.youtube.com/watch?v=_uyjp_HGZl4

We’d love to hear what you think about the product and vision, and ideas on how we can improve it. Thanks for taking the time to read this, we’re grateful to be posting here :)

Just pass this through openai clip and you would get a semantic search without much effort. For example this is for youtube videos - https://github.com/haltakov/natural-language-youtube-search

CLIP Video Search