mediapipe VS openpose

I was going to say that I like the MediaPipe Selfie Segmentation model for doing this sort of thing in a web page, but I've just noticed (when getting the GitHub link[1]) that Google have marked the code as legacy[2] ... no idea if the new solution is better/easier to use[3].

For what it's worth, my CodePen using the old model is here: https://codepen.io/kaliedarik/pen/PopBxBM

[1] - https://github.com/google/mediapipe/blob/master/docs/solutio...

[2] - "Attention: Thank you for your interest in MediaPipe Solutions. As of April 4, 2023, this solution was upgraded to a new MediaPipe Solution."

[3] - https://developers.google.com/mediapipe/solutions/vision/ima...

I found MediaPipe's Face Mesh and was impressed with how simple it was to get going, but it just gives you the landmark points and I've not gone any further yet.

We've trained ControlNet on a subset of the LAION-Face dataset using modified output from MediaPipe's face mesh annotator to provide a new level of control when generating images of faces.

Medipaipe library link: https://mediapipe.dev/

If you're looking for the differences in terms of how inference is done, I recommend you take a look at MediaPipe's source code. MediaPipe doesn't use raw code, but uses a "graph" instead (eg. pose_landmark_cpu.pbtxt), which can be visualised using MediaPipe Viz. I also used axinc-ai/ailia-models as the base (preprocessing, inference, postprocessing, etc...) which I further built upon (keypoint refinement, roi from keypoints, filtering / smoothing, etc...)

I thought of doing that but unfortunately medipipe requires a RGB input and performs better with it more on that here

OpenPose looks like what you're looking for, it seems to have plugins for Unity. I can't say I've used it though.

FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04 # https://hub.docker.com/r/nvidia/cuda ENV DEBIAN_FRONTEND=noninteractive # install the dependencies for building OpenPose RUN apt-get update && # The rest is ignored for brevity. RUN pip3 install --no-cache-dir # The rest is ignored for brevity. # install cmake, clone OpenPose and download models RUN wget https://cmake.org/files/v3.20/cmake-3.20.2-linux-x86_64.tar.gz && \ # The rest is ignored for brevity. WORKDIR /openpose/build RUN alias python=python3 && cmake -DBUILD_PYTHON=OFF -DWITH_GTK=OFF -DUSE_CUDNN=ON .. # Build OpenPose. Cudnn 8 causes memory issues this is why we are using base with CUDA 10 and Cudnn 7 # Fix for CUDA 10.0 and Cudnn 7 based on the post below. # https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1753#issuecomment-792431838 RUN sed -ie 's/set(AMPERE "80 86")/#&/g' ../cmake/Cuda.cmake && \ sed -ie 's/set(AMPERE "80 86")/#&/g' ../3rdparty/caffe/cmake/Cuda.cmake && \ make -j`nproc` && \ make install WORKDIR /openpose

Openpose: This is supposedly realtime (I assume on a gpu, 24fps?) and they provide training code

The pose tracking OP’s using relies on openpose - if you look at the GIFs in their readme, they appear to track individual fingers with fairly high resolution, so I’d imagine it would be fairly straightforward to map that to fret positions.

while I congratulate you for putting together a pipeline, I think it has to be said that the basis of this - the very powerful openpose - has a license that is not completely FOSS. It might be open source but is definitely not free as free beer. So if you consider using this for a fancy project (i.e. marketing an app), you have to get in touch with university of california first https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/LICENSE

Markerless motion capture technology (i.e. AI systems to track joint positions automatically from raw video) have come a long way

Caffe is a DL framework just like TensorFlow, PyTorch etc. OpenPose is a real-time person detection library, implemented in Caffe and c++. You can find the original paper here and the implementation here.

This is a good pose detection repo: https://github.com/CMU-Perceptual-Computing-Lab/openpose.