mpifileutils
go-find-duplicates
| mpifileutils | go-find-duplicates | |
|---|---|---|
| 4 | 3 | |
| 160 | 260 | |
| 0.6% | - | |
| 5.1 | 2.6 | |
| 21 days ago | 9 months ago | |
| C | Go | |
| BSD 3-clause "New" or "Revised" License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mpifileutils
-
Pigz: A parallel implementation of gzip for multi-core machines
If you ever run into the limitations of a single machine, dbz2 is also a fun little app for this sort of thing. You can run it across multiple machines and it'll automatically balance the workload across them.
https://github.com/hpc/mpifileutils/blob/master/man/dbz2.1
- MpiFileUtils: File utilities designed for scalability and performance
-
Go Find Duplicates: blazingly-fast simple-to-use tool to find duplicate files
If you want something that scales horizontally, dcmp from https://github.com/hpc/mpifileutils is an option.
- You can list a directory containing 8M files, but not with ls
go-find-duplicates
What are some alternatives?
fclones - Efficient Duplicate File Finder
fdupes - FDUPES is a program for identifying or deleting duplicate files residing within specified directories.
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
pigz - A parallel implementation of gzip for modern multi-processor, multi-core machines.
czkawka - Multi functional app to find duplicates, empty folders, similar images etc.
duphard - A simple utility to detect duplicate files and replace them with hard links.
dupd - CLI utility to find duplicate files
coreutils - Enhancements to the GNU coreutils (especiall head)
jdupes - A powerful duplicate file finder and an enhanced fork of 'fdupes'.