snapraid
MultiPar
snapraid | MultiPar | |
---|---|---|
86 | 21 | |
1,851 | 893 | |
- | - | |
6.7 | 8.2 | |
3 months ago | 22 days ago | |
C | C | |
GNU General Public License v3.0 only | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
snapraid
-
Storage software with the features of Unraid but runs on Debian with cli interface?
Would mergerfs and snapraid work for you? You'd sacrifice a disk to parity and run the parity calc manually, but you could set up a cron job for that.
- Data storage solution for "archival" purposes.
-
The Next Gen Database Servers Powering Let's Encrypt(2021)
Like most people on r/homelab, it started out with Plex. Rough timeline/services below:
0. Got a Synology DS413 with 4x WD Red 3TB drives. Use Playstation Media Server to stream videos from it. Eventually find some Busybox stuff to add various functionality to the NAS, but it had a habit of undoing them periodically, which was frustrating. I also experienced my first and (knock on wood) only drive failure during this time, which concluded without fanfare once the faulty drive was replaced, and the array repaired itself.
1. While teaching self Python as an Electrical Distribution Engineer at a utility, I befriended the IT head, who gave me an ancient (I think Nehalem? Quad-core Xeon) Dell T310. Promptly got more drives, totaling 7, and tried various OS / NAS platforms. I had OpenMediaVault for a while, but got tired of the UI fighting me when I knew how to do things in shell, so I switched to Debian (which it's based on anyway). Moved to MergerFS [0] + SnapRAID [1] for storage management, and Plex for media. I was also tinkering with various Linux stuff on it constantly.
1.1 Got tired of my tinkering breaking things and requiring troubleshooting/fixing (in retrospect, this provided excellent learning), so I installed Proxmox, reinstalled Debian, and made a golden image with everything set up as desired so I could easily revert.
1.2 A friend told me about Docker. I promptly moved Plex over to it, and probably around this time also got the *Arr Stack [2] going.
2. Got a Supermicro X9DRi-LN4F+ in a 2U chassis w/ 12x 3.5" bays. Got faster/bigger CPUs (E5-2680v2), more RAM, more drives, etc. Shifted container management to Docker Compose. Modded the BIOS to allow it to boot from a NVMe drive on a PCIe adapter.
2.1 Shifted to ZFS on Debian. Other than DKMS occasionally losing its mind during kernel upgrades, this worked well.
2.2 Forked [3] some [4] Packer/Ansible projects to suit my needs, made a VM for everything. NAS, Dev, Webserver, Docker host, etc. Other than outgrowing (IMO) MergerFS/SnapRAID, honestly at this point I could have easily stopped, and could to this day revert back to this setup. It was dead reliable and worked extremely well. IIRC I was also playing with Terraform at this time.
2.3 Successfully broke into tech (Associate SRE) as a mid-career shift, due largely (according to the hiring manager) to what I had done with my homelab. Hooray for hobbies paying off.
3. Got a single Dell R620. I think the idea was to install either pfSense or VyOS on it, but that never came to fruition. Networking was from a Unifi USG (their tiny router + firewall + switch) and 8-port switch, with some AC Pro APs.
4. Got two more R620s. Kubernetes all the things. Each one runs Proxmox in a 3-node cluster with two VMs - a control plane, and worker.
4.0.1 Perhaps worth noting here that I thoroughly tested my migration plan via spinning up some VMs in, IIRC, Digital Ocean that mimicked my home setup. I successfully ran it twice, which was good enough for me.
4.1 Played with Ceph via Rook, but a. disliked (and still to this day) running storage for everything out of K8s b. kept getting clock skew between nodes. Someone on Reddit mentioned it was my low-power C-state settings, but since that was saving me something like ~50 watts/node, I didn't want to deal with the higher power/heat. I landed on Longhorn [5] for cluster storage (i.e. anything that wasn't being handled by the ZFS pool), which was fine, but slow. SATA SSDs (used Intel enterprise drives with PLP, if you're wondering) over GBe aren't super fast, but they should be able to exceed 30 MBps.
4.1.1 Again, worth noting that I spent literally a week poring over every bit of Ceph documentation I could find, from the Red Hat stuff to random Wikis and blog posts. It's not something you just jump into, IMO, and most of the horror stories I read boiled down to "you didn't follow the recommended practices."
5. Got a newer Supermicro, an X11SSH-F, thinking that it would save power consumption over the older dual-socket I had for the NAS. It turned out to not make a big difference. For some reason I don't recall, I had a second X9DRi-LN4F+ mobo, so I sold the other one with the faster CPUs, bought some cheaper CPUs for the other one, and bought more drives for it. It's now a backup target that boots up daily to ingest ZFS snapshots. I have 100% on-site backups for everything. Important things (i.e. anything that I can't get from a torrent) are also off-site.
6. Got some Samsung PM863 NVMe SSDs mounted on PCIe adapters for the Dells, and set up Ceph, but this time handled by Proxmox. It's dead easy, and for whatever reason isn't troubled by the same clock skew issues as I had previously. Still in the process of shifting cluster storage from Longhorn, but I have been successfully using Ceph block storage as fast (1 GBe, anyway - a 10G switch is on the horizon) storage for databases.
So specifically, you asked what I do with the hardware. What I do, as far as my family is concerned, is block ads and serve media. On a more useful level, I try things out related to my job, most recently database-related (I moved from SRE to DBRE a year ago). I have MySQL and Postgres running, and am constantly playing with them. Can you actually do a live buffer pool resize in MySQL? (yes) Is XFS actually faster than ext4 for large DROP TABLE operations? (yes, but not by much) Is it faster to shut down a MySQL server and roll back to a previous ZFS snapshot than to rollback a big transaction? (often yes, although obviously a full shutdown has its own problems) Does Postgres suffer from the same write performance issue as MySQL with random PKs like UUIDv4, despite not clustering by default? (yes, but not to the same extent - still enough to matter, and you should use UUIDv7 if you absolutely need them)
I legitimately love this stuff. I could quite easily make do without a fancy enclosed rack and multiple servers, but I like them, so I have them. The fact that it tends to help my professional growth out at the same time is a bonus.
[0]: https://github.com/trapexit/mergerfs
[1]: https://www.snapraid.it
[2]: https://wiki.servarr.com
[3]: https://github.com/stephanGarland/packer-proxmox-templates
[4]: https://github.com/stephanGarland/ansible-initial-server
[5]: https://longhorn.io
-
Bitrot protection with BTRFS and Rsync
If you are using OpenMediaVault, checkout SnapRaid plugin.
-
Does this count?
I used drivepool for years with snapraid before I switched over to unraid. Nothing but good things to say about either program. Highly recommend both.
-
Merge/Raid HDD documentation
You can always use SnapRAID . there is no user interface, it is CLI. also you have to sync it manually. or set up a cronjob. you loose a hdd like unRaid or RAID5 but it gives you parity. then you could always use duplicati and backblaze business to make backups. it isnt as expensive as you would think for a homelab. the first back up might be a little much but then its pennies after that
-
Converting my old pc to a backup solution
As for the drives I'm thinking of grabbing a few from ServerPartDeals and upgrading my setup that uses DrivePool and snapRAID, but in Linux you would use mergerfs instead of DrivePool.
-
Thinking of switching from a 4 bay hardware RAID 5 to an 8 bay JBOD. Looking for opinions.
I myself prescribe to the teachings of the IronicBadger(Alex Kretzshmar) from the Self-Hosted podcast and (when I get one setup) intend to follow the guides on his site https://perfectmediaserver.com and use mergerfs to turn a JBOD to a single filesystem and use SnapRAID for redundancy.
-
WWYD? Help choosing b/w NAS, DAS or micro PC case?
I have that exact Sabrent 5 bay enclosure and I replaced it with this Orico 5 bay enclosure because the Sabrent's connection would fail while I ran snapraid sync, but the Orico has never failed me.
-
Just ordered 2 20TB CMR HDD’s and I’m extremely excited. What RAID method should I use?
With two drives? Just run them as-is. Once you get a third disk, set up https://www.snapraid.it. Then as you add more, just follow https://www.snapraid.it/faq#howmanypar to have the correct number of parity disks. Or don't, Plex content is probably easy to re-acquire if you need to so having redundancy or a backup isn't all that important.
MultiPar
-
Red Dead Redemption 2 PEDS_10 and rdr3.resident.rpf corrupted files and mismatch
but... depending on how much effort you want to spend fixing it I might be able to fix your current corrupt files, especially if the corrupt files are only mildly corrupt I could create a '.par2' file from mine (assuming you are on build 1436.28), then you would use MultiPar (i.e. https://github.com/Yutaka-Sawada/MultiPar/releases ) to scan your files and it will tell me how many blocks you need to fix it and then I could create recovery data with MultiPar. doing this might dramatically lower the amount of data I would have to upload as then you could use MultiPar which will fix your current corrupt files so they match mine identically.
-
WinRAR zero-day exploited since April to hack trading accounts
MultiPar is the continuation for Windows https://github.com/Yutaka-Sawada/MultiPar
And this for Linux https://github.com/animetosho/par2cmdline-turbo
-
It‘s been more than 2 years that animetosho plead uploaders to stop RAR‘in their Usenet uploads. What are Your thoughts on this in 2023?
* MultiPar supports in-situ repair, which bypasses this problem entirely
-
unRAID Under the premise of Parity Disk, how to specify to rebuild the specified directory or file?
To detect bitrot I use dynamix file integrity plugin, it compute Blake3 hash for each file, it is stored as extend attribute. You Can choose which share will be covered. For correction i then use par2 files using multipar https://hp.vector.co.jp/authors/VA021385/ https://github.com/Yutaka-Sawada/MultiPar (Windows only, par2cmdline for Linux. You could compile from source and add it to unraid if you want, i've done it for testing purpose)
- How does Multi Par works?
-
How to ensure file integrity?
And last but not less important - make regular parity files of your healthy important data, and store them somewhere. MultiPar is your best friend. Even if your data gets corrupt, with parity files you can restore it to it's former glory. Some limitations will apply, read the manual.
-
Is there a way to quickly create PAR2 files from the Windows context menu?
This is possible with MultiPar https://github.com/Yutaka-Sawada/MultiPar
-
I need to store a 1MB file for 20 years, would making thousands of copies on a 4.7gb DVD be enough?
More sanely, perhaps, get some M-Disc DVDs (if you can find real ones - apparently they're becoming just normal DVDs) and put one file on each, and then run https://github.com/Yutaka-Sawada/MultiPar on it to create parity files. Then store the file, the parity files, and the software you need (MultiPar) on the disc. To be safe, you probably also need to store the DVD burner you used to make it, as those are getting thin on the ground these days.
-
Best archival medium for 3-10 gig video files?
Note about par2: if one is using a windows system with a nvidia GPU this version of par2 is faster as it speaks CUDA.
-
Usenet vs Torrents for more results
For any file repairs use multipar, it's still being updated and has gpu acceleration. https://github.com/Yutaka-Sawada/MultiPar/releases
What are some alternatives?
mergerfs-tools - Optional tools to help manage data in a mergerfs pool
dvdisaster - A tool providing additional ECC protection for optical media (unofficial version)
Elucidate - Elucidate: A GUI to drive the SnapRAID command line (via .Net)
ParParGUI - GUI front-end to ParPar, a PAR2 creation tool
dupeguru - Find duplicate files
Nyuu - Flexible usenet binary posting tool
mergerfs - a featureful union filesystem
par2cmdline - Official repo for par2cmdline and libpar2
Rsnapshot - a tool for backing up your data using rsync (if you want to get help, use https://lists.sourceforge.net/lists/listinfo/rsnapshot-discuss)
HBBatchBeast - A free GUI application for HandBrake and FFmpeg/FFprobe with an emphasis on batch conversion (including recursive folder scans and folder watching) -Windows, macOS, Linux & Docker
snapraid-btrfs - Script for using snapraid with btrfs snapshots
NewsUP - Fully feature high performance binary usenet uploader/poster