CUB VS pytorch3d

If you still want to try it, the install.md shows how to download cub and set it for building it. I expect issues due to the versioning issues above.

Check out PyTorch3D. Are you trying to find the correct camera pose so the 3D object looks like the image? They have a tutorial for that :) Or is the goal to have a sort of classifier that tells you whether or not the 3D and 2D inputs belong together? Then I would probably go for some sort of contrastive learning. Separate encoders for image and 3D into a joint latent space, and then train by maximizing cosine similarity of matching entries in a batch (and minimizing for non-matching). Check out the clip embedding paper as a starting point, where it's done for text and images. Final question is, what architecture should you use for the 3D model. If you can have a variable number of vertices, something with attention like a small transformer is probably a good choice. Hope that helps :)

I am looking into pytorch3d(https://pytorch3d.org/) and also using the script directly.

import condacolab, torch, sys, skimage, matplotlib, imageio, plotly, cv2, black, flake8, facenet_pytorch from google.colab import drive drive.mount('/content/gdrive') !git clone https://github.com/Chinmayrane16/ReconNet-PyTorch !cp /content/gdrive/MyDrive/headsegmentation_final2.zip /content/gdrive/MyDrive/3DMM-Fitting-Pytorch.zip /content/ !cp /content/gdrive/MyDrive/Anaconda3.sh . !unzip -qq /content/3DMM-Fitting-Pytorch.zip !unzip -qq /content/ReconNet-PyTorch/images/all\ images/BSDS200.zip !mv /content/ReconNet-PyTorch/*.py /content/ !mkdir results !apt-get update -y !apt-get --purge remove "*cublas*" "cuda*" "nsight*" !wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin !mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 !wget https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb !dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb !apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub !apt-get install cuda-10-2 libtorch -y !apt autoremove -y !chmod 777 Anaconda3.sh !./Anaconda3.sh condacolab.install() !conda update conda !conda create -n pytorch3d python=3.9 !conda activate pytorch3d !conda install -c pytorch pytorch=1.9.1 torchvision cudatoolkit=10.2 !conda install -c fvcore -c iopath -c conda-forge fvcore iopath !conda install -c bottler nvidiacub !conda install jupyter !conda install pytorch3d -c pytorch3d pyt_version_str = torch.__version__.split("+")[0].replace(".", "") version_str="".join([f"py3{sys.version_info.minor}_cu", torch.version.cuda.replace(".",""), f"_pyt{pyt_version_str}"]) !pip3 install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html !export CUB_HOME=$PWD/cub-1.10.0 !pip3 install "git+https://github.com/facebookresearch/pytorch3d.git@stable" !rm -rf sample_data/ *.sh *.run

!pip3 install -q condacolab scikit-image matplotlib imageio plotly opencv-python black 'isort<5' flake8-bugbear flake8-comprehensions facenet-pytorch import condacolab, torch, sys from google.colab import drive drive.mount('/content/gdrive') !git clone https://github.com/Chinmayrane16/ReconNet-PyTorch !cp /content/gdrive/MyDrive/headsegmentation_final2.zip /content/gdrive/MyDrive/3DMM-Fitting-Pytorch.zip /content/ !cp /content/gdrive/MyDrive/Anaconda3.sh . !unzip -qq /content/3DMM-Fitting-Pytorch.zip !unzip -qq /content/ReconNet-PyTorch/images/all\ images/BSDS200.zip !mv /content/ReconNet-PyTorch/*.py /content/ !mkdir results !apt-get update -y !apt-get --purge remove "*cublas*" "cuda*" "nsight*" !wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin !mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600 !https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb !dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb !apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub !apt-get -y install cuda-10-2 !chmod 777 Anaconda3.sh !./Anaconda3.sh !rm -rf sample_data/ Anaconda3-2021.11-Linux-x86_64.sh cuda_10.2.89_440.33.01_linux.run condacolab.install() !conda update conda !conda create -n pytorch3d python=3.9 !conda activate pytorch3d !conda install -c pytorch pytorch=1.9.1 torchvision cudatoolkit=10.2 !conda install -c fvcore -c iopath -c conda-forge fvcore iopath !conda install -c bottler nvidiacub !conda install jupyter !conda install pytorch3d -c pytorch3d pyt_version_str = torch.__version__.split("+")[0].replace(".", "") version_str="".join([f"py3{sys.version_info.minor}_cu", torch.version.cuda.replace(".",""), f"_pyt{pyt_version_str}"]) !pip3 install --no-index --no-cache-dir pytorch3d -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/{version_str}/download.html !export CUB_HOME=$PWD/cub-1.10.0 !pip3 install "git+https://github.com/facebookresearch/pytorch3d.git@stable"

Sorry I don't have an answer to your question directly, but Facebook (Meta) are working on a library for 3D DL: https://github.com/facebookresearch/pytorch3d