facenet-pytorch VS OpenSeeFace

Hijacking this comment because i've been working nonstop on my project thanks to your suggestion. I'm now using this https://github.com/derronqi/yolov8-face for face detection and still the old face_recognition for encodings. I'm clustering with dbscan and extracting frames with ffmpeg with -hwaccel on. I'm planning to try this: https://github.com/timesler/facenet-pytorch as it looks like it would be the fastest thing avaiable to process videos? Keep in mind i need to perform encoding other than just detection because i want to use DBscan (and later also facial recognition, but this might be done separately just by saving the encodings). let me know if you have any other suggestions, and thanks again for your help

For now, I am trying to evaluate and get the accuracy of the FaceNet module. Like this example on facenet-pytorch, getting the accuracy relies on this file (pairs.txt) provided by the official site. Format description below:

with some python knowledge you can try a two step procedure: 1) extract a number of frames per second, for example five frames (images, i.e. still frames) per second using opencv or ffmpeg 2) Using facenet: detect faces in frames and then classify them by comparing each image to a known image of the person you are looking for.

It's generally better to split the task into a multiple tasks. First I'd want to detect and extract faces. There are a number of pretrained models that you could use for that, e.g. https://github.com/timesler/facenet-pytorch, https://github.com/opencv/opencv/tree/master/data/haarcascades. Once you've extracted faces, you can train a facial recognition using something like a siamese network as you normally would.

I got something working modifying OpenSeeFace and it's an option and I might try to rewrite it in something compiled, but I'd like to look at the other options first.

Lastly, face tracking is either built in or uses a plugin device. You would also use OSC to manipulate blendshapes. I'd take a look at Opens Face.

Th best/only options I found were basically the old standby of Wine running closed source applications. Even then the end result was rather incomplete... because may of the best tracking options are simply closed source. (Note: Vseeface does offer an open source library)

There are only a few libraries that come to mind but take a bit of work to get started. MediaPipe Unity Plugin has eye tracking with a whole lot of types of tracking(head, hands, body). OpenSeeFace has models that do head and eye tracking. This repo uses Unity's neural net inference library, Barracuda, to run a MediaPipe iris landmark model (I haven't personally tested this library). Not sure how to translate eye landmarks to the position a player is looking at in a screen though. Hopefully this list of libraries gets you on the right path!

OpenSeeFace is open source. If you are using it with VSeeFace, you can just replace the Binary folder with your own build.

It uses OpenSeeFace for facetracking and engine patches/vrm code from the V-Sekai team.