HR-VITON VS openpose

FROM tensorflow/tensorflow:1.15.5-gpu-py3 # Handle Nvidia public key update and update repositories for Ubuntu 18.x. #https://github.com/sangyun884/HR-VITON/issues/45 # reference: https://jdhao.github.io/2022/05/05/nvidia-apt-repo-public-key-error-fix/ RUN rm /etc/apt/sources.list.d/cuda.list RUN rm /etc/apt/sources.list.d/nvidia-ml.list RUN apt-key del 7fa2af80 # Additional reference: https://gitlab.com/nvidia/container-images/cuda/-/issues/158 RUN export this_distro="$(cat /etc/os-release | grep '^ID=' | awk -F'=' '{print $2}')" \ && export this_version="$(cat /etc/os-release | grep '^VERSION_ID=' | awk -F'=' '{print $2}' | sed 's/[^0-9]*//g')" \ && apt-key adv --fetch-keys "https://developer.download.nvidia.com/compute/cuda/repos/${this_distro}${this_version}/x86_64/3bf863cc.pub" \ && apt-key adv --fetch-keys "https://developer.download.nvidia.com/compute/machine-learning/repos/${this_distro}${this_version}/x86_64/7fa2af80.pub" # get the latest version of OpenCV RUN apt-get update && \ DEBIAN_FRONTEND=noninteractive \ apt-get install -y -qq \ wget git libopencv-dev RUN python -m pip install --upgrade pip && \ pip install matplotlib opencv-python==4.5.4.60 Pillow scipy \ azure-eventhub azure-eventhub-checkpointstoreblob-aio ipykernel WORKDIR /

Developing a garment to person virtual try on app, by far, the best results I could find is HR-VITON, another one is pasta-gan++(lower resolution, but should be acceptable after scale up). I have studied the papers of Gaugan

Open Pose, a multi-person keypoint detection library for body, face, hands, and foot estimation [10], is used for posing generated characters;

OpenPose

Perhaps something like OpenPose for pose estimation?

OpenPose looks like what you're looking for, it seems to have plugins for Unity. I can't say I've used it though.

You are correct. Open Pose has two keypoints for the eyes and two more for the ears. By saying were the ears are you automatically influence the angle of the head. You can see more about it on this github page. Just scroll a tiny bit and you can see a gif of the nodes overlapped on humans

FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04 # https://hub.docker.com/r/nvidia/cuda ENV DEBIAN_FRONTEND=noninteractive # install the dependencies for building OpenPose RUN apt-get update && # The rest is ignored for brevity. RUN pip3 install --no-cache-dir # The rest is ignored for brevity. # install cmake, clone OpenPose and download models RUN wget https://cmake.org/files/v3.20/cmake-3.20.2-linux-x86_64.tar.gz && \ # The rest is ignored for brevity. WORKDIR /openpose/build RUN alias python=python3 && cmake -DBUILD_PYTHON=OFF -DWITH_GTK=OFF -DUSE_CUDNN=ON .. # Build OpenPose. Cudnn 8 causes memory issues this is why we are using base with CUDA 10 and Cudnn 7 # Fix for CUDA 10.0 and Cudnn 7 based on the post below. # https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1753#issuecomment-792431838 RUN sed -ie 's/set(AMPERE "80 86")/#&/g' ../cmake/Cuda.cmake && \ sed -ie 's/set(AMPERE "80 86")/#&/g' ../3rdparty/caffe/cmake/Cuda.cmake && \ make -j`nproc` && \ make install WORKDIR /openpose

One of the best cam only libraries (no depth sensor) I've seen is openpose, I ran it through a 360 camera and it was able to track body, face, and fingers really well even with spherical distortion from the 360 cam. example 360