mediapipe VS tfjs-models

Medipaipe library link: https://mediapipe.dev/

If you're looking for the differences in terms of how inference is done, I recommend you take a look at MediaPipe's source code. MediaPipe doesn't use raw code, but uses a "graph" instead (eg. pose_landmark_cpu.pbtxt), which can be visualised using MediaPipe Viz. I also used axinc-ai/ailia-models as the base (preprocessing, inference, postprocessing, etc...) which I further built upon (keypoint refinement, roi from keypoints, filtering / smoothing, etc...)

I thought of doing that but unfortunately medipipe requires a RGB input and performs better with it more on that here

You might want to check out https://github.com/open-mmlab/mmhuman3d and https://github.com/google/mediapipe

This architecture was also used in the link referenced when bringing up alternative implementations:

https://github.com/google/mediapipe/issues/877#issuecomment-...

No clue if it will work on drawn images. It depends on what the model was trained on. The models github is a good place to get started on figuring that out: https://github.com/tensorflow/tfjs-models/tree/body-pix-v2.0.4/body-pix

You probably want to check out something like tensorflow https://github.com/tensorflow/tfjs-models where you can build and test your models.

All data to be used with a model needs to be prepared to be in the correct input format. In this case images needs to be turned into tensors. The exact format is usually documented but I found the underlying documentation for this model a bit lacking in details about the exact input/outputs but still useful for interpreting them https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection. Instead we can examine the model.json which is more explict. We can find a few keys that define the input and output:

The BlazePose software determines the position of the human body based on the camera image. For instance, a fitness app can automatically evaluate your technique, an online store can suggest suitable clothing styles, or games - create an avatar that repeats your actions. Model code is available on GitHub, there is also a demo in the browser.

Since the release of the latest version of the MediaPipe handpose detection machine learning model that allows the detection of multiple hands, I've had in mind to try to use it to create UIs, and here's the result of a quick prototype built in a few hours!

But what is MobileNet ?

In assets/video.html on lines 20-22, import TensorFlow.js and the coco-ssd model to detect "objects defined in the COCO dataset, which is a large-scale object detection, segmentation, and captioning dataset." It can detect 80 classes of objects. (SSD stands for Single Shot MultiBox Detection, kind-of like how YOLO stands for You Only Look Once). Read more about the model here on Google CodeLabs.

One of the models is called "HandPose" offers “Hand pose detection”. The description reads:

Load model: You need to load the model to be able to use it. In this case, we are using face landmarks detection model.

Sorry I must drop a a link to this for folks in this space… https://github.com/tensorflow/tfjs-models/tree/master/toxicity