OpenSeeFace VS kalidokit

I got something working modifying OpenSeeFace and it's an option and I might try to rewrite it in something compiled, but I'd like to look at the other options first.

Lastly, face tracking is either built in or uses a plugin device. You would also use OSC to manipulate blendshapes. I'd take a look at Opens Face.

Th best/only options I found were basically the old standby of Wine running closed source applications. Even then the end result was rather incomplete... because may of the best tracking options are simply closed source. (Note: Vseeface does offer an open source library)

There are only a few libraries that come to mind but take a bit of work to get started. MediaPipe Unity Plugin has eye tracking with a whole lot of types of tracking(head, hands, body). OpenSeeFace has models that do head and eye tracking. This repo uses Unity's neural net inference library, Barracuda, to run a MediaPipe iris landmark model (I haven't personally tested this library). Not sure how to translate eye landmarks to the position a player is looking at in a screen though. Hopefully this list of libraries gets you on the right path!

OpenSeeFace is open source. If you are using it with VSeeFace, you can just replace the Binary folder with your own build.

It uses OpenSeeFace for facetracking and engine patches/vrm code from the V-Sekai team.

I try to animate a ReadyPlayer Me avatar using ThreeJS and Kalidokit (or something else) with MediaPipe Hollisitc Pose. Here is a working JSFiddle :

kalidoface is made by /u/YeeMachineDev and seems to be fully open source https://github.com/yeemachine/kalidokit https://github.com/yeemachine/kalidoface-3d

In terms of software, what you linked used optical tracking. The underlying tech is commonly called machine/computer vision. You can develop your own tracking software, but there's some open source solutions, like OpenCV than make it faster to build apps. (You can use OpenCV with Python bindings to learn the basics somewhat quickly). What you linked uses Kalidokit, an open source solution using TensorFlow.js and Google's MediaPipe.

oh my apologies! may I suggest Kalidoface then? https://kalidoface.com/ free face and arm tracking! plus they have a very friendly discord for troubleshooting

[JS newbie here] I just discovered kalidokit, a vtuber JS app by yeemachine, and I'm trying to figure out how I could play with its look and feel. Do you have any recommendations on how to proceed? In short, I don't know what are the relevant files in the repo, how they are connected and how to run the app on a local machine. Any help would be very helpful! Thanks.

kalidokit