postgres-websockets VS postgresql-replicant

At work, we've finally replaced a large part of a custom (mostly-)web backend with PostgREST recently, and that's quite a relief: considerably less code to maintain in that project now, and that was a rather awkward code. Something akin to PostgREST's "Embedding with Top-level Filtering" [1] had to be provided for all the tables, with OpenAPI schema and a typed API (Haskell + Servant); I avoided manually writing it all down, but at the cost of poking framework internals, and maintainability suffered. It was particularly annoying that the code doesn't really do anything useful, except for standing between a database and an HTTP client, and simply mimics the database anyway. Whenever a change had to be introduced, it was introduced into the database, the backend, and the frontend simultaneously, so it wasn't even useful for some kind of compatibility.

Now PostgREST handles all that, and only a few less trivial endpoints are handled by a custom backend (including streaming, which I'm considering replacing with postgrest-websocket [2] at some point).

During the switch to PostgREST, the encountered minor issues were those with inherited tables (had to set a bunch of computed/virtual columns [3] in order to "embed" those), and with a bug on filtering using such relations (turned out it was an already-fixed regression [4], so an update helped). Also a couple of helper stored procedures (to use via /rpc/) for updates in multiple tables at once (many-to-many relationships, to edit entities along with their relationships, using fewer requests) were added (though the old custom backend didn't have that), the security policies were set from the beginning, the frontend was rewritten (which allowed to finally switch without adding more work), so it was only left to cleanup the backend.

Not using views, since as mentioned above, database changes usually correspond to frontend changes, and the API doesn't have to be that stable yet.

Happy with it so far.

[1] https://postgrest.org/en/stable/api.html#embedding-with-top-...

[2] https://github.com/diogob/postgres-websockets

[3] https://postgrest.org/en/stable/api.html#computed-virtual-co...

[4] https://github.com/PostgREST/postgrest/issues/2530

You can use a library like https://github.com/agentultra/postgresql-replicant or similar to hook in a data plane and use PostgREST as a data plane.

Your business logic works on the event stream that comes from the WAL.

I wrote postgresql-replicant [0] in order to do this. It’s not a novel idea. It lets you read the logical replication stream.

On top of this library you can do all sorts of things: replicate data to Kafka or SQS, write a web socket server to publish changes to clients, etc.

[0] https://github.com/agentultra/postgresql-replicant

https://github.com/agentultra/postgresql-replicant — stream delta changes from a PostgreSQL database. Not a novel concept or implementation but this library doesn’t exist in Haskell and that’s a shame because Haskell has lots of great libraries for working with streams.

https://github.com/agent ultra/DataVersion — type-directed data migration library. I needed to migrate some data in a document database that had no schema with years of data in it to a format that had a schema. I leveraged some other stuff to achieve it but this library was a key piece to being confident that the transformations were correct and complete. Migrated the database and backfill it with a Lambda function using this library.

Those are just the most recent ones.

On my stream at https://twitch.tv/agentultra I've been working on a library to stream changes out of a PostgreSQL database. We've had to figure out how to use low-level libpq bindings, concurrency primitives, serializing and deserializing binary messages, etc. You can find the library here: https://github.com/agentultra/postgresql-replicant

You can star the repo and report issues at: https://github.com/agentultra/postgresql-replicant -- new contributors are welcome! I'm still working out a contribution guide, it's still super early on the project. I can be found here and on the FP Slack (https://fpchat-invite.herokuapp.com/) and on the FP Zulip (https://funprog.zulipchat.com)