compsize
duperemove
compsize | duperemove | |
---|---|---|
8 | 16 | |
325 | 662 | |
- | - | |
3.0 | 9.2 | |
4 months ago | about 1 month ago | |
C | C | |
GNU General Public License v3.0 or later | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
compsize
-
BTRFS Transparent Compression
The tool compsize can give you info about the compression state of your data. For example:
-
Is anyone else having problems with the zfs AUR packages? Specifically, zfs-dkms?
The obvious example is the Raid5/Raid6 implementation which is officially not recommended for current use; though that's not something I would use anyways. But I've also found that stuff like listing the compression ratio for volumes requires a separate program, unlike zfs where it's built-in.
- compsize: btrfs: find compression type/ratio on a file or set of files
-
Space allocation
If you want to diagnose where you think there may be space used up that you don't quite understand, then btrfs fi du and third party tools btdu and compsize are useful.
-
Bringing Bcachefs to Linux Mainline
I'm going to add my N=1 anecdata if no one minds:
I've been running full Btrfs on my workstation since December 2021 with 0 issues (kernel 5.10). It survived multiple hard reboots and power offs, Ryzen 5000 CPUs seems to lose power every now and then on Linux (it's a known issue for years now that no one has figured out yet).
Having compression is great. compsize reports I've saved 33GB on my /home subvolume so far (https://github.com/kilobyte/compsize). I don't use snapshots, I just backup the whole thing.
Obviously this doesn't mean that in the future Btrfs won't eat my data, but so far it hasn't and I'm happy.
- Using BTRFS compression on my Steam Deck, defragging and compsize
-
Adding options to fstab?
You can also use this tool to find out what the compression ratio of the data is ;).
- ZFS 0.8.6 doesn't boot with 5.10 kernel on Debian Sid
duperemove
-
fdupes: Identify or Delete Duplicate Files
Very useful for identifying files that may need to get deduplicate or that can be removed entirely. Unfortunately, I don't think this will also find identical directories.
If deleting files isn't what you want, I'd suggest looking into deduplicating tools.
ZFS has its own de duplicator built in, which is nice. It should just deduplicate files and individual extents of files by itself once you enable it. Probably not a good idea on very write-heavy disks, but it's an option.
Other file systems with extent level deduplication can use https://github.com/markfasheh/duperemove to not only deduplicaye files, but also deduplicate individual extents. This can be very useful for file systems that store a lot of duplicate content, like different WINE prefixes. For filesystems without extent deduplication, duperemove should try hard linking files to make them take up practically no disks space.
-
Bcachefs Merged into the Linux 6.7 Kernel
ZFS now has reflink support, which doesn't require lots of RAM, but isn't done automatically while writing. You need to run something like https://github.com/markfasheh/duperemove
- ZFS 2.2.0 (RC): Block Cloning merged
- Craziest thing I ever used SQLite for: partial file deduplication
-
Deduplication on EXT4
Then duperemove
-
What is a wineprefix and should I make a new one every time I add a new game to lutris?
Filesystems like Btrfs and XFS have support for deduplication, you can use a program like duperemove to save space.
-
File deduplication report?
Maybe you could use a file deduplication instead of a block based? https://github.com/markfasheh/duperemove
-
Anyone running Bees? Or deduping data some other way?
If not bees, do you run other programs for deduping? I see jdupes has support for BTRFS, https://github.com/jbruchon/jdupes, and also duperemove, https://github.com/markfasheh/duperemove.
-
Is switching to BTRFS useful for my use-cases?
It's a good filesystem, I use it with a special setup that needs a filesystem with snapshots. It's been stable for me, I run a duperemovehttps://github.com/markfasheh/duperemove) occasionally and that's about all the maintenance it needs.
-
With Proton being as good as it is now, do we still need separate prefixes for every game?
With Btrfs or XFS you can easily deduplicate the data with tools like duperemove, potentially saving a lot of space if you've installed many small games.
What are some alternatives?
btrfdeck - This repo will get you from using ext4 on your Steam Deck's microSD card, to btrfs.
bees - Best-Effort Extent-Same, a btrfs dedupe agent
btdu - sampling disk usage profiler for btrfs
jdupes - A powerful duplicate file finder and an enhanced fork of 'fdupes'.
archzfs - Package and repository sources for ZFS on Arch Linux
dduper - Fast block-level out-of-band BTRFS deduplication tool.
zfs - OpenZFS on Linux and FreeBSD
btrfs - WinBtrfs - an open-source btrfs driver for Windows
zfs-dkms-aur-pkg - NOTE: Use this patch at your own risk. Testing a patch for the zfs-dkms Arch Linux AUR package
rmlint - Extremely fast tool to remove duplicates and other lint from your filesystem
bees-docker - docker container for zygo/bees
rdfind - find duplicate files utility