shutter VS fastdup

I mean 100% synthetic data. When you generate random layouts you know where you are placing the objects you are interested in detecting with your model (e.g. keys, values). If you have no common layout then I’d go creative and try to cover the most common! I have a repo I used to generate fake documents, it’s not maintained anymore but maybe you find it useful.

Visualizing your dataset (especially large ones) in a low-dimensional embedding space can tell you a lot about the patterns and clusters in your dataset.

We recently release a notebook showing how you can visualize your dataset using DINOv2 models by running it on your CPU.

Yes! No GPUs needed.

We used it to find clusters of similar images, duplicates, and outliers in a subset of the LAION dataset

Try it on your own dataset:

Colab notebook - https://colab.research.google.com/github/visual-layer/fastdup/blob/main/examples/dinov2_notebook.ipynb

GitHub repo - https://github.com/visual-layer/fastdup

Run 1M images from the LAION dataset through the DINOv2 model and cluster the images using a free tool - fastdup.

Another option could be fastdup (https://github.com/visual-layer/fastdup) which is probably most helpful for analysis type objectives.

I recently stumbled upon fastdup a tool that lets you gain insights from a large image/video collection.

There is this fastdup framework which supposedly allows you to find duplicates and similar images. i haven't used it though

I came across fastdup recently https://github.com/visual-layer/fastdup