annuaire-entreprises-sirene-api VS postgresql-replicant

One of the project which list all the data for entreprise addresses uses Postgrest (open data SIRENE de l'INSEE).

Full fledge project implemented, interesting use of PostgREST

https://github.com/etalab/annuaire-entreprises-sirene-api

You can use a library like https://github.com/agentultra/postgresql-replicant or similar to hook in a data plane and use PostgREST as a data plane.

Your business logic works on the event stream that comes from the WAL.

I wrote postgresql-replicant [0] in order to do this. It’s not a novel idea. It lets you read the logical replication stream.

On top of this library you can do all sorts of things: replicate data to Kafka or SQS, write a web socket server to publish changes to clients, etc.

[0] https://github.com/agentultra/postgresql-replicant

https://github.com/agentultra/postgresql-replicant — stream delta changes from a PostgreSQL database. Not a novel concept or implementation but this library doesn’t exist in Haskell and that’s a shame because Haskell has lots of great libraries for working with streams.

https://github.com/agent ultra/DataVersion — type-directed data migration library. I needed to migrate some data in a document database that had no schema with years of data in it to a format that had a schema. I leveraged some other stuff to achieve it but this library was a key piece to being confident that the transformations were correct and complete. Migrated the database and backfill it with a Lambda function using this library.

Those are just the most recent ones.

On my stream at https://twitch.tv/agentultra I've been working on a library to stream changes out of a PostgreSQL database. We've had to figure out how to use low-level libpq bindings, concurrency primitives, serializing and deserializing binary messages, etc. You can find the library here: https://github.com/agentultra/postgresql-replicant

You can star the repo and report issues at: https://github.com/agentultra/postgresql-replicant -- new contributors are welcome! I'm still working out a contribution guide, it's still super early on the project. I can be found here and on the FP Slack (https://fpchat-invite.herokuapp.com/) and on the FP Zulip (https://funprog.zulipchat.com)