Hierarchical-Localization VS nerfmm

You can try hierarchical localization, it's pretty memory efficient since it only brings up relevant point clouds instead of the entire mapped pointed when you're computing poses.

Sounds like this is a 3D to 2D correspondence estimation problem. So is it correct that you are trying find the pose of the object based on seen 2D images? First you need to define a canonical reference frame for the object. This object reference frame is essentially glued to the object and you want to estimate the object to camera frame transformation matrix which will give you the pose of the object relative to how you are viewing it from a given frame. To achieve this, most literature use some form of 3D to 2D feature correspondence search from which a transformation matrix is obtained using projective geometry. Features like SIFT features can be used to find correspondences between features seen in the 2D image and features in the 3D object. This is also an active area of research in computer vision and the state of the art uses learned deep features. You can check out https://github.com/cvg/Hierarchical-Localization which is the State-of-the-Art in camera 6DOF pose estimation from known 3D models of the world. For your scenario, you just need to define the object coordinate system and you can obtain the pose if you know the object to camera transformations. You should also first look into the classical approaches which use some variants of PNP + RANSAC algorithm to find 2D to 3D correspondences. Since you also know the relative poses of the cameras, you can also do refinement like bundle adjustment to better predict your 2D to 3D correspondences. Let me know if you find any good tutorials or resources online.

Different traditional methods can also be helped by specific ML tasks such as: (a) initial 2D bounding box detection to limit region of 3D pose estimation (b) edge detection (like HED: https://arxiv.org/abs/1504.06375) (c) training on a photogrametry model for more robust retrieval and matching in changing scale and light (like HLoC: https://github.com/cvg/Hierarchical-Localization)

I have been very successful with these kind of image pairs using SuperPoint+SuperGlue. The authors‘ work is awesome and code is available here: hloc

I am looking for some advice on a problem I am running into using this localization algorithm.