backy2 VS wal-e

Overbuilt and OTT? Sure... but this works fantastically for my use case. I have current backups of everything except my media library because of the size of it; my VM's are all backed up to my Synology nightly using Backy2, my application data gets dumped to that same Synology NAS nightly as well, and all of that also gets synced to Glacier deep storage once a week using Duplicity. I'm going to be adding a new ZFS array later in the year to replace my Synology and hopefully I'll build it out with enough storage to take my media library as well.

It's also been fun for discovery of new things... new tools and use cases. Being able to use Cephfs is great but also being able to leverage it as native S3 buckets is awesome. Learning how to manage snapshots both in RBD images (for my VM's) and Cephfs is cool, and developing my own scripts to snapshot and replicate critical data to my Synology has been rewarding. There's also some pretty cool tools out there even without being as well supported as ZFS like backy2 for backing up RBD images... again to my Synology with a fun little script.

I've been a DataHoarder for a while, but only a modest ~10TB or so. I finally had the space to set up a rack and some servers, and am setting up a Ceph cluster with a ton of old disks I've accumulated over the years, totaling upwards of 20TB. I would like to still have an offsite and preferably offline backup for this data though, but backing up 20+ TB of data to a single drive is obviously off the table. Is there any other alternative to just deploying another Ceph cluster offsite? I don't want to use cloud storage due to the costs, and I also very much prefer to keep all my data under my own physical control. I was looking at Backy2 for the actual extraction of data and writing it to a destination, but that doesn't seem to support idempotent writes (i.e. take one full object and place it on a single drive). I could theoretically combine drives via LVM, but without additional redundancy (I would probably use raid 1 for that) losing one drive would be disastrous, and I am trying to avoid having to add additional redundancy for backups, considering the main ceph cluster will already have 3 copies of the data on it. I also am wondering if I should avoid using Ceph for the backups, since then all my eggs would be in the Ceph basket so to speak. I would love some advice from some of the folks with larger hoards and how you make backups. Thank you!

Have you tried reaching out to the dev on GitHub? I have before and was able to get some bugs ironed out. https://github.com/wamdam/backy2

See the GitHub: https://github.com/wal-e/wal-e

Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :)

Seems to be obsolete due to a lack of interest and contributions.

I believe the idea originally started with Wal-E project (https://github.com/wal-e/wal-e) since then there were many improvements to it and AWS no doubt has their own proprietary solution.

Hope this helps someone out there.

https://github.com/wal-e/wal-e

Running your own postgres server on AWS is not difficult. You'll want to get familiar with EBS and look into using a tool like wal-e to manage your WAL logs and ensure yuor ability to recover from the various exciting failure modes that Amazon makes available to you.

These read-replicas leveraged WAL segments created by the primary database; however, we didn’t want to bog down the primary database with each replica by reading directly from production. To circumvent this issue, we leverage wal-e, a tool that performs continuous archiving of PostgreSQL WAL files and base backups, and read replicas restored from s3 or gcs versus reading directly from the primary database.

Have a look at barman, pgBackrest or for maybe WAL-G or WAL-E

I am using the following docker image https://github.com/docker-library/postgres. I want the ability to make regular backups to my S3 bucket. PITR is not necessary but a nice to have. I've tried to use https://github.com/wal-e/wal-e but I've not found something that works very well with docker.