Our great sponsors
-
-
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
-
-
-
-
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Hey HN, I’m an engineer at Supabase [0] and one of the creators of this demo. My team and I have been working hard to bring developers the next version of Supabase Realtime.
The current version of Realtime [1] is a Change Data Capture (CDC) server for a PostgreSQL database that broadcasts changes via WebSockets to authorized subscribers. It’s written in Elixir/Phoenix.
The server utilizes PostgreSQL’s logical replication functionality, which writes database changes to Write-Ahead Logging (WAL) segment files, and a replication slot, responsible for managing and retaining WAL files.
Database changes are polled from WAL by the server using PostgreSQL’s replication function pg_logical_slot_get_changes and changes converted to JSON objects using the wal2json [2] extension by setting it as the output plugin.
Security is enforced through two checks - each check ensures only authorized client subscribers are sent database changes. The first check validates a JWT that is sent by clients subscribing to database changes. This JWT must contain an existing database role and optional claims, both of which can be referenced in Row Level Security (RLS) policies. Every valid client subscription is then inserted into the realtime.subscription table with an assigned UUID, database role, and claims. The second check calls the realtime.apply_rls SQL function from Write Ahead Log Realtime Unified Security (WALRUS) utility lib [3]. This function takes the database changes, executes a prepared statement to verify if the database role and claims have SELECT permissions on the changes, and outputs an array of authorized UUIDs. Then, the server finds all the subscribers whose UUIDs are in that array and broadcasts the changes to them.
The next version of Supabase Realtime will offer three features: Broadcast, Presence, and Extensions.
Broadcast, our Pub/Sub offering, can be used to pass ephemeral data from client to client such as cursor movements. This runs on a distributed cluster of nodes built on top of Phoenix PubSub + Channels.
Presence, can be used for tracking online/offline users and their state. This is built into Phoenix, and uses replicated state across a cluster using an Observe-Remove-Set-Without-Tombstones (ORSWOT) CRDT [4] which prefers adds over removes when resolving conflicts.
Extensions, are a way for the community to add additional functionality to take advantage of the WebSocket infrastructure. We have converted the existing Change Data Capture system to an extension that supports connecting to multiple customer databases (multi-tenancy). Other possible extensions include listening to other databases like MySQL and getting stock market events server-side [5], then broadcasting them to connected clients.
This demo is built using a Supabase project, Supabase Realtime, and Next.js and deployed on 20 Fly [6] nodes located around the world. You can find an introduction and walkthrough of the demo here [5].
Supabase Realtime is entirely open source and you can find the demo code here [7]. Once we have stabilized the release we will add it to the self-hosted offering [8]. This demo is a way to highlight the upcoming features and gather feedback/ideas.
Feel free to ask me anything and let me know what you think!
Of course! You can check out the demo code here: https://github.com/supabase/realtime/tree/multiplayer.
> CRDTs
definitely. it's one of the primary use-cases we had in mind when developing this, and something we've wanted to offer for a long time. I'm happy to see it popping up early in the comments. We wouldn't offer our own CRDTs, but Realtime can be a nice transport layer for other CRDT implementations (which can then be serialised and persisted to your database)
> YJS powered
I'm also glad to see you're using Yjs - it's very cool. We hope that this implementation can be another Yjs Provider[0] if Kevin is onboard with that. Once that's implemented, you would be able to use it with all the same bindings (i.e. y-monaco).
> write-up soon as a guide for Python developers moving to Elixir
Awesome initiative!
> ETS as a KV/document store to hold user and application state and then reacting to changes in to that the way you are here
This is actually pretty interesting. I can't speak to ETS but Mnesia has replication and you can expose the replication log using something like https://github.com/emqx/mria. I've only had a cursory look at this so I could be wrong about its capabilities but it would be an awesome extension to the new Realtime if possible.
> finding the building blocks of modern applications (database, auth, functions, presence, realtime subscriptions), making them easy to use, and then sharing the source code.
Great observation!
> I’ve learned a ton just from cruising around supabase GitHub.
Glad to hear it!
> Can you say which of these new components will be open sourced?
All of these components are open source and licensed under Apache License v2.0.
> There are some other features (e.g. function hooks) that are also closed-source at the moment.
I actually worked on the initial implementation of function hooks. We've actually already open sourced both the client (see: https://github.com/supabase/supabase/tree/88bcef911669595428...) and the pg_net extension it requires (see: https://github.com/supabase/pg_net).
> Is Supabase heading for an “open core” model?
I don't think so. We want to continue to open source our projects under either MIT (client libs) and Apache License v2.0 (server libs).