Track-Anything VS lama

I was doing rotoscoping for a silhouette of a girl dancing in front of a building, then I saw this amazing tool: https://github.com/gaomingqi/Track-Anything

My question is, how can we modernize this pipeline? We've experimented a bit with the new SAM-based track-anything tool, and it seems promising, but we actually don't want to "track anything", we only want to track fishes. What would you do in 2023, to extract tracks of one specific class of object from long video datasets? I'm hoping for any advice at all.

The Track-Anything tool already implements this

Segment Anything for Video - Track Anything! 🤖 With this tool, you can automatically isolate objects, make edits using inpainting, and track objects with precision. It's a game-changer for creative projects. Even though it does not work well with the shadows yet, we expect a rapid evolution of these technologies. Github : https://github.com/gaomingqi/Track-Anything)

I would like to know if there are any guidelines to follow when adapting an existing Python project for my own use-case. Specifically, I want to customize the output of the Track-Anything project by incorporating my own processing steps. However, I do not want to import the entire codebase. Rather, I only want to import the minimum amount of code necessary to produce the same output with object tracking, without having to reimplement functions that are already available.

Here is the GitHub: https://github.com/gaomingqi/track-anything

You could try to use ControlNet inpaint+lama locally, but results aren't as good in my experience. Or you could try local install of lama directly, but the setup process isn't very smooth.

You may be able to remove the actor with lama. https://github.com/advimman/lama

LaMa: Resolution-robust Large Mask Inpainting with Fourier Convolutions (Apache-2.0 license) Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, Victor Lempitsky (Samsung Research and EPFL)

You could try using LaMA: https://github.com/saic-mdal/lama .

Task 1: 30$ Write a script to prepare data for https://github.com/saic-mdal/lama. You should be able to prove on video that your script works properly after i give you some data and you train and there's some acceptable output (not perfect, since i know how ml works).

I have some interest in image inpainting, but the recent paper introduced by Samsung - https://github.com/saic-mdal/lama seems like it reached a stage where there isn't much room that we can add for novelty. I am quite interested but not confident if I can come up with a novel idea that can do better than the proposed model, especially if they trained the model on multiple GPUs for days, which is something that I don't have.

Code: https://github.com/saic-mdal/lama