wal2json VS walrus

To illustrate the above, let’s look at an example, using wal2json for convenience. An update event looks something like this:

You could also use tools in conjunction with logical replication to stream changes, like debezium or wal2json.

I've done CDC before for mysql and postgres, in both cases, rolled my own 200~ loc python scripts. No experience with DMS. Debezium, as I understand, needs a kafka topic to stream to, so wasn't workable for us. Rolling our own proved to be fairly trivial. For postgres, there's wal2json https://github.com/eulerto/wal2json

I was researching for the whole day and found out https://github.com/eulerto/wal2json and postgres bult-in `pgoutput` plugin help me to catch the changes.

I don't think this is something for "rails" to do per se, and more the underlying DB. replication tools to capture a "change log". I have use postgres Write Ahead Logs (WAL) shipped to s3 in json format https://github.com/eulerto/wal2json

[0] https://supabase.com [1] https://github.com/supabase/realtime [2] https://github.com/eulerto/wal2json [3] https://github.com/supabase/walrus [4] https://gsd.di.uminho.pt/members/cbm/ps/delta-crdt-draft16ma... [5] https://supabase.com/blog/2022/04/01/supabase-realtime-with-... [6] https://fly.io [7] https://github.com/supabase/realtime/tree/multiplayer [8] https://github.com/supabase/supabase/tree/master/docker

wal2json Output Plugin that converts WAL output to JSON objects [Open Source] pg_recvlogical Postgres app that can consume update stream [Out-of-the-box with Postgres] decoderbufs Output Plugin that delivers data as protobuf [Open Source, Used in Debezium]

Wen Bo from the Supabase Realtime team here!

We actually use https://github.com/supabase/walrus to get the database changes and we're planning on moving to a Rust worker (https://github.com/supabase/walrus/tree/worker/worker) for better performance especially in the RLS-enabled use case.

I promised José that I would check out Postgrex.ReplicationConnection and we might add it in to our Supabase Realtime some time in the future.

[0] https://supabase.com [1] https://github.com/supabase/realtime [2] https://github.com/eulerto/wal2json [3] https://github.com/supabase/walrus [4] https://gsd.di.uminho.pt/members/cbm/ps/delta-crdt-draft16ma... [5] https://supabase.com/blog/2022/04/01/supabase-realtime-with-... [6] https://fly.io [7] https://github.com/supabase/realtime/tree/multiplayer [8] https://github.com/supabase/supabase/tree/master/docker

[Supabase cofounder] thanks for the kind words.

We're about to wrap up Row Level Security on our real-time APIs too[0] - soon you'll be able to use Policies universally on all APIs.

[0] https://github.com/supabase/walrus

We are adding this to the database now: https://github.com/supabase/walrus