rdfind
cargo-crev
rdfind | cargo-crev | |
---|---|---|
16 | 55 | |
883 | 2,034 | |
- | 1.7% | |
4.1 | 7.7 | |
about 1 month ago | 4 days ago | |
C++ | Rust | |
GNU General Public License v3.0 or later | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rdfind
- Rdfind: A utilty to find duplicate files, delete them or replace with hardlinks
-
Self hosted, web gui, file duplication scanner
I use rdfind for this.
-
Is there a Mac app that will allow me to recursively go through thousands of folders, calculate the total folder size, then compare against all other folder sizes, and if the size is identical, delete the newer one?
rdfind is available for macOS; I've been using it on linux: https://github.com/pauldreik/rdfind
-
Deduplication on EXT4
You can use rdfind to find all duplicates in your experiments dir and replace files with hardlinks. This way files will occupy disk space only once and all inode references will be to the same disk location.
- How do I show non-duplicate files across 2 drives?
-
Pip and cargo are not the same
I use rdfind to deal with this: https://github.com/pauldreik/rdfind
- Backing Up Data: Tips/Advice for Tons of Unorganized Data and Duplicate Files from Multiple Sources
-
This has probably happened to all of us at least once
Yeah, I periodically download the full drives and just deduplicate with rdfind hardlinking identical files.
- AMD/Xilinx Vivado rant
-
recommends for de-duplication?
I use rdfind on my Linux NAS. https://github.com/pauldreik/rdfind
cargo-crev
-
Hard disk LEDs and noisy machines
In other cases it may be more documented, such as Golangs baked-in telemetry.
There should be better ways to check these problems. The best I have found so far is Crev https://github.com/crev-dev/crev/. It's most used implementation is Cargo-crev https://github.com/crev-dev/cargo-crev, but hopefully it will become more required to use these types of tools. Certainty and metrics about how many eyes have been on a particular script, and what expertise they have would be a huge win for software.
-
Rust Without Crates.io
The main problem the author is talking about is actually about version updates, which in Maven as well as crates.io is up to each lib's author, and is not curated in any way.
There's no technical solution to that, really. Do you think Nexus Firewall can pick up every exploit, or even most? How confident of that are you, and what data do you have to back that up? I don't have any myself, but would not be surprised at all if "hackers" can easily work around their scanning.
However, I don't have a better approach than using scanning tools like Nexus, or as the author proposes, use a curated library repository like Debian is doing (which hopefully gets enough eyeballs to remain secure) or the https://github.com/crev-dev/cargo-crev project (manually reviewed code) also mentioned. It's interesting that they mention C/C++ just rely on distros providing dynamic libs instead which means you don't even control your dependencies versions, some distro does (how reliable is the distro?)... I wonder if that could work for other languages or if it's just as painful as it looks in the C world.
-
I don't care about cookies” extension bought by Avast, users jump ship
For instance, the worst company imaginable may be in charge of software that was once FOSS, and they may change absolutely nothing about it, so it should be fine. However, if a small update is added that does something bad, you should know about it immediately.
The solution seems to be much more clearly in the realm of things like crev: https://github.com/crev-dev/cargo-crev/
Wherein users can get a clear picture of what dependencies are used in the full chain, and how they have been independently reviewed for security and privacy. That's the real solution for the future. A quick score that is available upon display everytime you upgrade, with large warnings for anything above a certain threshold.
-
I think there should be some type of crates vertification especially the popular ones?
The metrics on crates.io are a useful sniff test, but ultimately you need to review things yourself, or trust some contributors and reviewers. Some projects, like cargo crev or cargo vet can help with the process.
-
[Discussion] What crates would you like to see?
You can use cargo-geiger or cargo-crev to check for whether people you trusted (e.g. u/jonhoo ) trust this crate.
-
Pip and cargo are not the same
There is a similar idea being explored with https://github.com/crev-dev/cargo-crev - you trust a reviewer who reviews crates for trustworthiness, as well as other reviewers.
- greater supply chain attack risk due to large dependency trees?
-
Why so many basic features are not part of the standard library?
[cargo-crev](https://github.com/crev-dev/cargo-crev) looks like a good step in the right direction but not really commonly used.
-
“You meant to install ripgrep”
'cargo crev' makes this kind of workflow possible: https://github.com/crev-dev/cargo-crev
-
Difference between cargo-vet and cargo-crev?
The crev folks themselves are no fans of PGP but need a way to security identify that you are in fact the review author, so that's where the id generation comes in. Ultimately crev is just a bunch of repos with text files you sign with IDs. The nice property is that you can chain these together into a web of trust and it's unfortunate that vet doesn't just use the same signed files on repos model as a foundation because even if they don't trust anyone else, we could turn around and trust them.
What are some alternatives?
fdupes - FDUPES is a program for identifying or deleting duplicate files residing within specified directories.
crates.io - The Rust package registry
jdupes - A powerful duplicate file finder and an enhanced fork of 'fdupes'.
stackage - Stable Haskell package sets: vetted consistent packages from Hackage
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
crates.io-index - Registry index for crates.io
fclones - Efficient Duplicate File Finder
serde - Serialization framework for Rust
dupeguru - Find duplicate files
cargo-msrv - 🦀 Find the minimum supported Rust version (MSRV) for your project
kindfs - Index filesystem into a database, then easily make queries e.g. to find duplicates files/dirs, or mount the index with FUSE.
Rustup - The Rust toolchain installer