slowpokefs VS keppel

One thing I saw was someone logging all system calls their software generated and comparing that between runs. They'd been bitten by some minor library upgrade that had changed directory scanning which caused huge multiples of normal latency which on an SDD wasn't noticeable but on an HDD went from trivial to insane.

That was handy because it didn't actually involve testing on a slow device but mostly showed what we needed to know.

There are tools like https://github.com/schoentoon/slowpokefs which try to make your fast storage act like a HDD which could be better for black-box testing.

Even if you have your prod database on RDS, you might be able to spin up a local instance for testing. My Postgres-using applications have a script that sets up and runs a local Postgres instance for the duration of the test. It takes about 2 seconds on first run, and less than noticeable time when the DB has already been initialized.

https://github.com/sapcc/keppel/blob/master/testing/with-pos...