lama VS openpose

You could try to use ControlNet inpaint+lama locally, but results aren't as good in my experience. Or you could try local install of lama directly, but the setup process isn't very smooth.

You may be able to remove the actor with lama. https://github.com/advimman/lama

LaMa: Resolution-robust Large Mask Inpainting with Fourier Convolutions (Apache-2.0 license) Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, Victor Lempitsky (Samsung Research and EPFL)

You could try using LaMA: https://github.com/saic-mdal/lama .

Task 1: 30$ Write a script to prepare data for https://github.com/saic-mdal/lama. You should be able to prove on video that your script works properly after i give you some data and you train and there's some acceptable output (not perfect, since i know how ml works).

I have some interest in image inpainting, but the recent paper introduced by Samsung - https://github.com/saic-mdal/lama seems like it reached a stage where there isn't much room that we can add for novelty. I am quite interested but not confident if I can come up with a novel idea that can do better than the proposed model, especially if they trained the model on multiple GPUs for days, which is something that I don't have.

Code: https://github.com/saic-mdal/lama

Open Pose, a multi-person keypoint detection library for body, face, hands, and foot estimation [10], is used for posing generated characters;

OpenPose

Perhaps something like OpenPose for pose estimation?

OpenPose looks like what you're looking for, it seems to have plugins for Unity. I can't say I've used it though.

You are correct. Open Pose has two keypoints for the eyes and two more for the ears. By saying were the ears are you automatically influence the angle of the head. You can see more about it on this github page. Just scroll a tiny bit and you can see a gif of the nodes overlapped on humans

FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04 # https://hub.docker.com/r/nvidia/cuda ENV DEBIAN_FRONTEND=noninteractive # install the dependencies for building OpenPose RUN apt-get update && # The rest is ignored for brevity. RUN pip3 install --no-cache-dir # The rest is ignored for brevity. # install cmake, clone OpenPose and download models RUN wget https://cmake.org/files/v3.20/cmake-3.20.2-linux-x86_64.tar.gz && \ # The rest is ignored for brevity. WORKDIR /openpose/build RUN alias python=python3 && cmake -DBUILD_PYTHON=OFF -DWITH_GTK=OFF -DUSE_CUDNN=ON .. # Build OpenPose. Cudnn 8 causes memory issues this is why we are using base with CUDA 10 and Cudnn 7 # Fix for CUDA 10.0 and Cudnn 7 based on the post below. # https://github.com/CMU-Perceptual-Computing-Lab/openpose/issues/1753#issuecomment-792431838 RUN sed -ie 's/set(AMPERE "80 86")/#&/g' ../cmake/Cuda.cmake && \ sed -ie 's/set(AMPERE "80 86")/#&/g' ../3rdparty/caffe/cmake/Cuda.cmake && \ make -j`nproc` && \ make install WORKDIR /openpose

One of the best cam only libraries (no depth sensor) I've seen is openpose, I ran it through a 360 camera and it was able to track body, face, and fingers really well even with spherical distortion from the 360 cam. example 360