pgbadger VS minion

It might also help to use pgbadger or something similar to process your postgres logs and see whether some event is aligned with your outages.

Interestingly, when a place does get to the point where the single instance has capacity issues (after upgrading to EPYC and lots of flash drives) then other non-obvious stuff shows up too.

For example, at one place just over a year ago they were well into this territory. One of weird problems for them was with pgBadger's memory usage (https://github.com/darold/pgbadger). That's written in perl, which doesn't seem to go garbage collection well. So even on a reporting node with a few hundred GB's of ram, it could take more than 24 hours to do a "monthly" reporting run.

There wasn't a solution in place at the time I left, so they're probably still having the issue... ;)

First of all I used a profiler (pgbadger and netdata) to figure out where the lags were coming from. I then tried the usual stuff (increasing shared_buffers, max_wal_size, min_wal_size from their ultra low defaults), but the biggest performance gain came from moving the database from eMMC to a mechanical hard drive :-D

Look into PgBadger (a log parser/analyser): https://github.com/darold/pgbadger

What finally drove me over the edge was writing some code for a client using the Minion job queue. The queue is solid, but it creates new database connections, thus ensuring that it can’t see anything in your database transactions. I figured out a (hackish) solution, but I was tired of hackish solutions.

Since Minion relies on PostrgeSQL the actual implementation was very simple. Just a few lines of Perl to spawn spare processes and one small change in the SQL query used to dequeue jobs.