pgBackRest VS wal-e

This tutorial explains how to backup PostgreSQL database using pgBackRest and S3.

This isn't theoretical; many companies do PostgreSQL async 1:N physical replication, by using e.g. https://pgbackrest.org/ to have the primary push WAL segment files (a.k.a. "the last n milliseconds of packets" in the write-ahead log) as objects to S3, and then to have all read-replicas fetch from S3 and replay.

> You could do even better if you out-of-band signal the readiness so you do not need to poll while idle.

S3 and its clones have "object lifecycle notifications", where you can be informed by a push-based mechanism whenever a new object is put into the bucket.

But — what do you have to do, to get these notifications?

Subscribe to a message queue that S3 puts them into.

I haven't explored the territory in awhile but for bare-metal, you can't go wrong with Percona Distribution, which includes pgBackRest and a minimal web-ui. No one ever got fired for using Percona, etc.

Check out pgBackRest

I found this approach pretty cool in that regard: https://github.com/pgbackrest/pgbackrest

Something like PG barman or pg backrest could be good for you on the Postgres side.

Backups are limited. These days, pgbackrest is the go-to backup solution for PostgreSQL, and having used it I am very impressed so far. It provides full backups, differential, and incremental, as well as archiving of WAL segments for point in time recovery. It allows great flexibility in schedules and destinations for backups, how long to keep backups for, how many full backups. For example, you can have backups made to a local disk, and other backups to an external S3-compatible bucket, each with their own settings and schedules (e.g., scheduled via cron).

See the GitHub: https://github.com/wal-e/wal-e

Unmaintained would’ve made more sense to say, but the maintainer choose the words “obsolete” so I took those. :)

Seems to be obsolete due to a lack of interest and contributions.

I believe the idea originally started with Wal-E project (https://github.com/wal-e/wal-e) since then there were many improvements to it and AWS no doubt has their own proprietary solution.

Hope this helps someone out there.

https://github.com/wal-e/wal-e

Running your own postgres server on AWS is not difficult. You'll want to get familiar with EBS and look into using a tool like wal-e to manage your WAL logs and ensure yuor ability to recover from the various exciting failure modes that Amazon makes available to you.

These read-replicas leveraged WAL segments created by the primary database; however, we didn’t want to bog down the primary database with each replica by reading directly from production. To circumvent this issue, we leverage wal-e, a tool that performs continuous archiving of PostgreSQL WAL files and base backups, and read replicas restored from s3 or gcs versus reading directly from the primary database.

Have a look at barman, pgBackrest or for maybe WAL-G or WAL-E

I am using the following docker image https://github.com/docker-library/postgres. I want the ability to make regular backups to my S3 bucket. PITR is not necessary but a nice to have. I've tried to use https://github.com/wal-e/wal-e but I've not found something that works very well with docker.