realtime VS wal2json

You can LISTEN/NOTIFY. Or you can use logical replication and a custom subscriber.[1] Supabase uses the latter.[2]

[1]: https://www.postgresql.org/docs/current/logical-replication....

[2]: https://github.com/supabase/realtime

Supabase Realtime is great for building collaborative applications. You can receive database changes over websockets, store and synchronize data about user presence, and broadcast any data to clients via "channels".

We use :telemetry to collect usage data per tenant for Supabase Realtime.

We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.

Yo :D This is what Supabase Realtime does!

https://github.com/supabase/realtime

Spin up a Supabase database and then subscribe to changes with WebSockets.

You can play with it here once you have a db: https://realtime.supabase.com/inspector/new

Every project is a Postgres database, wrapped in a suite of tools like Auth, Storage, Edge Functions, Realtime and Vectors, and encompassed by API middleware and logs.

Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.

Also, what was the reason not to go with Gevent?

I believe #2 was the main driver for the supabase team to build their real-time component: https://github.com/supabase/realtime

Background/announcement: https://supabase.com/blog/supabase-realtime-multiplayer-gene...

As a minor note, one of the linked articles talks about having used RethinkDB for its changefeeds and I made a mental note a bit back that if I ever want that supabase's realtime ( https://github.com/supabase/realtime ) provides something rather like that atop Postgres and I should try that before doing anything clever.

To illustrate the above, let’s look at an example, using wal2json for convenience. An update event looks something like this:

You could also use tools in conjunction with logical replication to stream changes, like debezium or wal2json.

I've done CDC before for mysql and postgres, in both cases, rolled my own 200~ loc python scripts. No experience with DMS. Debezium, as I understand, needs a kafka topic to stream to, so wasn't workable for us. Rolling our own proved to be fairly trivial. For postgres, there's wal2json https://github.com/eulerto/wal2json

I was researching for the whole day and found out https://github.com/eulerto/wal2json and postgres bult-in `pgoutput` plugin help me to catch the changes.

I don't think this is something for "rails" to do per se, and more the underlying DB. replication tools to capture a "change log". I have use postgres Write Ahead Logs (WAL) shipped to s3 in json format https://github.com/eulerto/wal2json

[0] https://supabase.com [1] https://github.com/supabase/realtime [2] https://github.com/eulerto/wal2json [3] https://github.com/supabase/walrus [4] https://gsd.di.uminho.pt/members/cbm/ps/delta-crdt-draft16ma... [5] https://supabase.com/blog/2022/04/01/supabase-realtime-with-... [6] https://fly.io [7] https://github.com/supabase/realtime/tree/multiplayer [8] https://github.com/supabase/supabase/tree/master/docker

wal2json Output Plugin that converts WAL output to JSON objects [Open Source] pg_recvlogical Postgres app that can consume update stream [Out-of-the-box with Postgres] decoderbufs Output Plugin that delivers data as protobuf [Open Source, Used in Debezium]