MiVOS VS detectron2

Mivos, you'll get assisted by a deep learning model https://github.com/hkchengrex/MiVOS

We've been building a ton of infrastructure to process a lot of video data very quickly at Sieve, and video object segmentation is a cool task a lot of companies have used us for. Models like MiVOS and PointRend have made it possible for companies like RunwayML to build these ML-based video editing features like green screen. We wrote about how you might go about building this yourself in a day.

Which is built on https://github.com/facebookresearch/detectron2

Click to Learn more...

Detectron2 (GitHub: https://github.com/facebookresearch/detectron2) is a Facebook AI Research library with state-of-the-art object detection and segmentation algorithms in PyTorch.

That would be awesome. But I think that Detectron only provides RCNN, but I could be wrong. At least the model zoo on github looks like it.

I could fine-tune a Detectron2 model a few months ago using PyTorch and MPS backend [1]. I'd be interested if it's working yet.

https://github.com/facebookresearch/detectron2/issues/4342

I wanted to have a few choices getting localization into image search (index and search time). I immediately thought of using a region proposal network (rpn) from mask-rcnn to create patches that can also be indexed and searched (and add the localisation). I figured it might be somewhat agnostic to classes. I did not want to use mmdetection or detectron2 due to their dependencies and just getting the rpn was not worth it. I was encouraged by the PyTorch native implementations of detection/segmentation models but ended up finding yolox the best.

You should already be able to use the code for object detection data if you do the outlier detection based on feature embeddings rather than predictions! You just need to get embeddings out of your object detection network, as demonstrated here for example: https://detectron2.readthedocs.io/en/latest/tutorials/models.html#partially-execute-a-model https://github.com/facebookresearch/detectron2/issues/5

detrex was built on detectron2 and we used the powerful LazyConfig system for more flexible syntax and cleaner config files.

We've been building a ton of infrastructure to process a lot of video data very quickly at Sieve, and video object segmentation is a cool task a lot of companies have used us for. Models like MiVOS and PointRend have made it possible for companies like RunwayML to build these ML-based video editing features like green screen. We wrote about how you might go about building this yourself in a day.