U-2-Net VS MiDaS

I use a finetuned custom Stable Diffusion model in combination with a style embedding for the characters for image generation and U²-Net for background removal.

Hi :) I am app/web developer and new to AI. Currently, I am making photo app which can segment all the things in image. I've used meta's segment anything. I've got all the masks but the boundary of masks are very bumpy. So I've tried rembg which uses u2net(salient object detection) and pymatting together. Do I have to use pymatting separately after getting segment from segment anything to improve boundary quality of my segmented output ?

Cool, thanks for sharing. It might be worth clearly attributing the models you're using, and maybe add a models/license file with the U2net license, since that license is different to the one you're using for your project, and since you're distributing the models.

Background Removal - I'd use u2net which has a model that's specifically trained on people vs backgrounds. If that didn't work, maybe DIS which is the newer version or rembg. These are pretty easy to get running I found.

u2net_human_seg (download, source): A pre-trained model for human segmentation.

Optimized – AI tweak Image background remover is now faster and enables trained model (onnx) swapping Revamp the python engine for background remover. Should be running faster than the previous build. Also added was the ability to replace the pre-trained ONNX model by the user themselves. https://github.com/xuebinqin/U-2-Net

-Added AI Background remover based on U2Net AI framework for Image projector. Check the Image projection "AI Tweaks" dropdown to toggle between u2net standard, u2netp – portrait, u2net-human_seg, u2net_cloth_seg, or silueta as the background remover AI model. As you can guess from the names, each of the models excels for different image subjects for background removal. For example, the portrait model is good for human portraits, cloth seg for clothing subjects and so on. Default to CUDA (Nvidia) processor, if you have an AMD card or would like to use CPU as the processor, untick the CUDA checkbox. Go here if you want to know more about the mechanics of U2Net -> https://github.com/xuebinqin/U-2-Net

Also checkout https://github.com/xuebinqin/U-2-Net. They have a new version in this repo: https://github.com/xuebinqin/DIS

There, the author has the same goal as you do, and has a train.py and instructions. You can reach out to the author and ask questions either in the issues section or perhaps email directly. Many times people are very helpful when you show interest in their work. The neural network it is based on (U2-net) is very easy to get running by the way, and has lots of use cases: https://github.com/xuebinqin/U-2-Net

really any segmentation model could work. "salient object detection" is well suited for "i have a single, obvious subject that I want to isolate from the background". This is the model I had in mind, but it wouldn't have to be this necessarily: https://github.com/xuebinqin/U-2-Net

The checkpoint below uses MiDaS, which returns the inverse depth map, so we have to invert it back to get a comparable depth map.

Hi, I have made use of the KITTI dataset for this, and yes it depends on objects of know sizes. Here I have defined the following classes: Car, Van, Truck, Pedestrian, Person_sitting, Cyclist, Tram, Misc, or DontCare and the predictions are pretty accurate for those classes. Even if it's not the same class, it still recognizes the object since I have made use of the coco names dataset here and that is used along with YOLO for object detection. And there are several already implemented projects that make use of deep learning models trained on 2D datasets to predict 3D distance. This was one of my inspirations for this project: https://blogs.nvidia.com/blog/2019/06/19/drive-labs-distance-to-object-detection/ Furthermore, there are well-documented and researched papers like DistYOLO or MiDaS that makes use of deep learning for depth estimation

-Added AI image depth generator Create your own depth map image at a click of a button. Using the awesome MIDAS3.1 https://github.com/isl-org/MiDaS as the backend and the model "dpt_beit_large_512" for the highest quality depth map. Video and GIF depth map generators coming out next together with the Depth movie player feature.

It uses MiDaS. That extension may be the most accessible way to use it at home. IDK.

Here's the project homepage https://github.com/isl-org/MiDaS

The problem came up after MiDaS updated to version V3_1 on Dec 24th. Although the fix works fine, with the new version there are many changes, which for me produces slightly different results. I would like to able to produce results like before. I still clone the MiDaS repo, but then set it back to the last commit before the changes in december, which is 66882994a432727317267145dc3c2e47ec78c38a.

try: from midas.dpt_depth import DPTDepthModel except: if not os.path.exists('MiDaS'): gitclone("https://github.com/isl-org/MiDaS.git") gitclone("https://github.com/bytedance/Next-ViT.git", f'{PROJECT_DIR}/externals/Next_ViT') if not os.path.exists('MiDaS/midas_utils.py'): shutil.move('MiDaS/utils.py', 'MiDaS/midas_utils.py') if not os.path.exists(f'{model_path}/dpt_large-midas-2f21e586.pt'): wget("https://github.com/intel-isl/DPT/releases/download/1_0/dpt_large-midas-2f21e586.pt", model_path) sys.path.append(f'{PROJECT_DIR}/MiDaS')

Cool. The repo for MiDaS is here. https://github.com/isl-org/MiDaS You can see that they partially trained the model on 3D movies Here's a list of the movies that were used to train it. I wonder if they'll be training a MiDaS v 4.0 as things have moved on quite a bit since it was released in Apr 2021?

We present a stand-alone implementation of our Merging Operator. This new repo allows using any pair of monocular depth estimations in our double estimation. This includes using separate networks for base and high-res estimations, using networks not supported by this repo (such as Midas-v3), or using manually edited depth maps for artistic use. This will also be useful for scientists developing CNN-based MDE as a way to quickly apply double estimation to their own network. For more details please take a look here.

I'll make the feature to export only the depthmap .png file. If you need the depthmap .png right now you can use the MiDaS python script.