automatic-video-processing VS deep_sort

I'm one of the creators of Sieve, and I'm looking for general thoughts on this problem.

https://sievedata.com seems very promising, a search engine for videos, with specific tags, sounds like a very good idea.

I'd like the same for all my photos and videos: that would be so much easier to find specific pictures by keywords

The reason I ask is because I'm working on something called Sieve. We're basically making it really easy for any software developer to process and understand video content. This includes applications from home security, to pet monitoring, baby monitoring, sports analytics, and media understanding.

Hey everyone! I'm the creator of Sieve, an API for AI-based video understanding. One application we're starting to support in beta is tracking player / object movements, speed, etc in video games. All you do is push video to our API, which we then process, after which you can search + query using API calls. We're starting by supporting a few popular games like League of Legends, Dota 2, CSGO, and Overwatch. Here are the docs.

For context I'm the founder of a company called Sieve which is starting to work with some of these tools to automatically gauge things like attentiveness and facial expressions by automatically analyzing the video. Would be interesting to hear what you as users actually want.

P.S. I am potentially very bias because I'm working on Sieve which is trying to work with these applications.

Hey everyone! I recently started building Sieve, a really easy way for devs to understand video content. We've just started to work with quite a few video editing tools / companies (both online and offline ones) after having primarily focused on real-world applications like security, supply chain, and general media.

Hey everyone! I’m one of the creators of Sieve. I posted about it here a while back and thought I'd share that r/MachineLearning can now try it for free :)

Hi HN, we’re Mokshith and Abhi from Sieve (https://sievedata.com). We’re building an API that lets you add video search to internal tools or customer applications, instantly. Sieve can process 24 hours of video in less than 10 minutes, and makes it easy to search video by detected objects / characteristics, motion data, and visual similarity. You can use our models out of the box, or plug-in your own model endpoints into our infrastructure. Models can mean any software that produces output given an image.

Every industry from security, to media, supply chain, construction, retail, sports, and agriculture is being transformed by video analytics—but setting up the infrastructure to process video data quickly is difficult. Having to deal with video ingestion pipelines, computer-vision model training, and search functionality is not pretty. We’re building a platform that takes care of all of this so teams can focus on their domain-expertise, building industry-specific software.

We met in high school, and were on the robotics team together. It was our first exposure to computer vision, and something we both deeply enjoyed. We ended up going to UC Berkeley together and worked on computer vision at places like Scale AI, Niantic, Ford, NVIDIA, Microsoft, and Second Spectrum. We were initially trying to solve problems for ourselves as computer vision developers but quickly realized the unique problems in video having to do with cost, efficiency, and scale. We also realized how important video would be in lots of verticals, and saw an opportunity to build infrastructure which wouldn’t have to be rebuilt by a fullstack dev at any company again.

Let’s take the example of cloud software for construction which might include tons of features from asset trackers to rental management and compliance checks. It doesn’t make sense for them to build their own video processing for telematics—the density and scale of video make this a difficult task. A single 30 FPS camera generates over 2.5M frames within a day of recording. Imagine this across thousands of cameras and many weeks of footage—not to mention the actual vertical-specific software they’re building for end users.

Sieve takes care of everything hard about processing and searching video. Our API allows you to process and search video with just two API calls. We use filtering, parallelization, and interpolation techniques to keep costs low, while being able to process 24 hours of video in under 10 minutes. Users can choose from our pre-existing set of models, or use their own models with our video processing engine. Our pricing can range anywhere from $0.08-$0.45 per minute of video processed based on the models clients are interested in and usage volume. Our FAQ page (https://sievedata.com/faq) explains these factors in more detail.

Our backend is built on serverless functions. We split each video into individual chunks which are processed in parallel and passed through multiple layers of filters to determine which chunks are “important”. We’re able to algorithmically ignore parts of video which are static, or change minimally, and focus on the parts that contain real action. We then run more expensive models on the most “important” parts of video, and interpolate results across frames to return information to customers at 30 FPS granularity. Our customers simply push signed video URLs to our platform, and this happens automatically. You can then use our API to query for intervals of interest.

We haven’t built an automated sign up flow yet because of our focus on the core product, but we still wanted to give all of you the chance to try Sieve on your own videos for free. You’ll be emailed a personal, limited-access API key.

Try it out: https://sieve-data.notion.site/Trying-Sieve-s-Video-Search-4...

Visual dashboard demo: https://www.youtube.com/watch?v=_uyjp_HGZl4

We’d love to hear what you think about the product and vision, and ideas on how we can improve it. Thanks for taking the time to read this, we’re grateful to be posting here :)

Similari is a Rust/Python framework aimed at building sophisticated tracking systems. With Similari, you can develop highly efficient parallelized SORT, DeepSORT, and other sophisticated multiple-object tracking engines.

I'm doing a Python personal project where I'm trying to use YOLOv8 and DeepSORT to detect vehicles from a car's dash cam footage. I succeeded in using YOLOv8 to output the correct bounding boxes by processing each camera frame. However, I tried to add on DeepSORT code, but it made the detection accuracy significantly worse. I'm pretty sure I need to train my own "feature extractor" for DeepSORT to create a new .pb file. I got this information from the deep_sort GitHub link: https://github.com/nwojke/deep_sort. I tried to find resources to do this but they are pretty scarce. Has anyone had experience with this problem?

The feature model well that looks like it is at that domain you mentioned but why not instead of using this repo use the original authors repo

nwojke/deep_sort

I was implementing the Kalman filter for bounding boxes during the last two days. As an inspiration source, I looked at the Python3 Kalman filter implementation that is used in the DeepSORT algorithm and uses NumPy and SciKit under the hood, so it's pretty efficient because all the operations are run inside FFI.

There's tons of work out there when it comes to object tracking such as DeepSort. We've worked to build simpler, more efficient solutions in-house though. Then past that, it's a matter of treating everything in the video as an object (including the whole frame), tracking it, and saving it in a no-SQL DB such that it's easy to query in this way.

Right now, the way we're thinking about it is to turn videos into something that works with the structure of a database like MongoDB. Everything in an image or a video is an object (even the frame itself is an object with a large bounding box), and each of these objects has some attributes and can be tracked over time with some form of object tracking. Given that the objects are tracked, they can each basically be returned as a time-series of each of the attributes associated with that object.

Deep Sort

I was wondering if I could use the same annotated data(in YOLO format) for the training of the tracker as well. I took a look at the original repo for DeepSORT, and it does mention the training using cosine metric learning, but I could not seem to understand how to replicate that for my own dataset(they show us how to do it for the MARS and Market1501 datasets).

DeepSORT is one of the best trackers https://github.com/nwojke/deep_sortIt requires an object detector tho, like YOLO https://pjreddie.com/darknet/yolo/