yugabyte-db VS realtime

It can be done like https://github.com/yugabyte/yugabyte-db/ has.

This is a workaround, not supported. I've opened the following issue to get it solve in the YugabyteDB deployment: https://github.com/yugabyte/yugabyte-db/issues/19389

FROM almalinux:8 as build RUN dnf -y update &&\ dnf groupinstall -y 'Development Tools' # get YugabyteDB sources ARG YB_TAG=2.18 RUN git clone --branch ${YB_TAG} https://github.com/yugabyte/yugabyte-db.git WORKDIR yugabyte-db # install dependencies and compilation tools RUN dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm RUN dnf -y install epel-release libatomic rsync python3-devel cmake3 java-1.8.0-openjdk maven npm golang gcc-toolset-12 gcc-toolset-12-libatomic-devel patchelf glibc-langpack-en ccache vim wget python3.11-devel python3.11-pip clang ncurses-devel readline-devel libsqlite3x-devel RUN mkdir /opt/yb-build RUN chown "$USER" /opt/yb-build # Install Python 3 RUN alternatives --remove-all python3 RUN alternatives --remove-all python RUN alternatives --install /usr/bin/python python /usr/bin/python3.11 3 RUN alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 3 # add #include "pg_yb_utils.h" to src/postgres/src/pl/plpython/plpy_procedure.c RUN sed -e '/#include "postgres.h"/a#include "pg_yb_utils.h"' -i src/postgres/src/pl/plpython/plpy_procedure.c # if using python > 3.9 remove #include and #include from src/postgres/src/pl/plpython/plpython.h RUN sed -e '/#include /d' -e '/#include /d' -i src/postgres/src/pl/plpython/plpython.h # add '--with-python', to python/yugabyte/build_postgres.py under the configure_postgres method RUN sed -e "/'\.\/configure',/a\ '--with-python'," -i python/yugabyte/build_postgres.py # Build and package the release RUN YB_CCACHE_DIR="$HOME/.cache/yb_ccache" ./yb_build.sh -j$(nproc) --clean-all --build-yugabyted-ui --no-linuxbrew --clang15 -f release RUN chmod +x bin/get_clients.sh bin/parse_contention.py bin/yb-check-consistency.py RUN YB_USE_LINUXBREW=0 ./yb_release --force WORKDIR / RUN mv /yugabyte-db/build/yugabyte*.tar.gz /yugabyte.tgz

www.yugabyte.com

In a previous post, I published an example with PostgREST on YugabyteDB. Here is another one: pREST opens a REST API to PostgreSQL. YugabyteDB is a PostgreSQL-compatible Open-Source Distributed SQL database. It adds horizontal scalability to applications built for PostgreSQL. Let's see how it integrates with pREST.

There is still work in progress in FerretDB, like Create primary key index for _id automatically #1384 . If a Primary Key is added with with the ID, it will become the sharding key in YugabyteDB. Another optimization will be to avoid reading information_schema.columns which is slow on YugabyteDB (the catalog must be shared by all nodes). This will be optimized on YugabyteDB (#7745). If it is still a scalability issue, there's also the possibility to fork the PostgreSQL handler to optimize it for YugabyteDB. All this is open source 🤩 YugabyteDB and FerretDB are Apache License 2.0

I was building a new version of YugabyteDB vagrant box with packer and virtual box. Because we (Yugabyte) have a new preview release out.

The sst_dump command in version 2.15.3.2 is not yet updated to handle packed rows. I created an issue for it: [DocDB] sst_dump does not recognise packed rows and displays 'Schema packing not found: 0: .'.

YugabyteDB

You can LISTEN/NOTIFY. Or you can use logical replication and a custom subscriber.[1] Supabase uses the latter.[2]

[1]: https://www.postgresql.org/docs/current/logical-replication....

[2]: https://github.com/supabase/realtime

We use :telemetry to collect usage data per tenant for Supabase Realtime.

We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.

Yo :D This is what Supabase Realtime does!

https://github.com/supabase/realtime

Spin up a Supabase database and then subscribe to changes with WebSockets.

You can play with it here once you have a db: https://realtime.supabase.com/inspector/new

Every project is a Postgres database, wrapped in a suite of tools like Auth, Storage, Edge Functions, Realtime and Vectors, and encompassed by API middleware and logs.

Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.

Also, what was the reason not to go with Gevent?

I believe #2 was the main driver for the supabase team to build their real-time component: https://github.com/supabase/realtime

Background/announcement: https://supabase.com/blog/supabase-realtime-multiplayer-gene...

How does supabase not qualify as open source?

Their stack is primarily comprised of other independent open source projects. The one component that isn't is their "realtime" server that serves updates from postgres' WAL over websockets, but that is open sourced[0] under Apache 2.0. From my understanding the primary part that has not been open sourced is their database browser / web UI. There are plenty of alternative management tools for postgres though. As you can export your database what else would you need to ensure your portability and independence?

Granted they make their docs fairly opaque for trying to self host. Presumably to encourage you to just use their hosted service. Hosting open sourced projects seems like a very ecosystem friendly way of monetizing.

[0] https://github.com/supabase/realtime

There is a few second delay when starting the subscriptions.

Yes, some portion of their backend is Elixir/Phoenix: https://github.com/supabase/realtime

but most of their stack, frontend and backend, is Next.js, and not the Rails-type fullstack way: https://github.com/supabase/supabase

They're all similar flavors of producing realtime results - which take similar, but different, methods to their approach.

My understanding (please feel free to correct me if I'm wrong):

- Supabase Realtime uses WAL.

- Hasura Streaming Subscriptions uses an append-only query (could also use WAL).

- Hasura Live Queries uses interval polling, refetching, and multiplexing.

- Supabase uses Postgres RLS for authorization, while Hasura uses an internal RLS system which composes queries (which allows for features like the multiplexing above).

- All 3 use websockets for their client communication.

Supabase Realtime

https://github.com/supabase/realtime#introduction

https://supabase.com/docs/guides/realtime

Hasura Subscriptions

https://github.com/hasura/graphql-engine/blob/master/archite...

https://github.com/hasura/graphql-engine/blob/master/archite...