realtime
litestream
Our great sponsors
realtime | litestream | |
---|---|---|
54 | 164 | |
6,418 | 9,700 | |
1.6% | - | |
9.0 | 7.5 | |
2 days ago | 7 days ago | |
Elixir | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
realtime
-
A Technical Dive into PostgreSQL's replication mechanisms
You can LISTEN/NOTIFY. Or you can use logical replication and a custom subscriber.[1] Supabase uses the latter.[2]
[1]: https://www.postgresql.org/docs/current/logical-replication....
-
Unpacking Elixir: Observability
We use :telemetry to collect usage data per tenant for Supabase Realtime.
We do this for rate limiting but it also makes it very easy for us to attach a listener (https://github.com/supabase/realtime/blob/main/lib/realtime/...) which ships these (per second) aggregates to BigQuery (via Logflare), which then the billing team can aggregate further to display and actually bill people with.
-
All the ways to capture changes in Postgres
Yo :D This is what Supabase Realtime does!
https://github.com/supabase/realtime
Spin up a Supabase database and then subscribe to changes with WebSockets.
You can play with it here once you have a db: https://realtime.supabase.com/inspector/new
-
Supabase Local Dev: migrations, branching, and observability
Every project is a Postgres database, wrapped in a suite of tools like Auth, Storage, Edge Functions, Realtime and Vectors, and encompassed by API middleware and logs.
-
Writing a chat application in Django 4.2 using async StreamingHttpResponse
Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.
Also, what was the reason not to go with Gevent?
-
How to Listen to Database Changes Using Postgres Triggers in Elixir
I believe #2 was the main driver for the supabase team to build their real-time component: https://github.com/supabase/realtime
Background/announcement: https://supabase.com/blog/supabase-realtime-multiplayer-gene...
-
From Plex to Jellyfin Media Server
How does supabase not qualify as open source?
Their stack is primarily comprised of other independent open source projects. The one component that isn't is their "realtime" server that serves updates from postgres' WAL over websockets, but that is open sourced[0] under Apache 2.0. From my understanding the primary part that has not been open sourced is their database browser / web UI. There are plenty of alternative management tools for postgres though. As you can export your database what else would you need to ensure your portability and independence?
Granted they make their docs fairly opaque for trying to self host. Presumably to encourage you to just use their hosted service. Hosting open sourced projects seems like a very ecosystem friendly way of monetizing.
-
Supabase Subscriptions Just Got Easier
There is a few second delay when starting the subscriptions.
-
Finding Relationships Between Ruby’s Top Packages and Their Dependencies
Yes, some portion of their backend is Elixir/Phoenix: https://github.com/supabase/realtime
but most of their stack, frontend and backend, is Next.js, and not the Rails-type fullstack way: https://github.com/supabase/supabase
-
Streaming data in Postgres to 1M clients with GraphQL
They're all similar flavors of producing realtime results - which take similar, but different, methods to their approach.
My understanding (please feel free to correct me if I'm wrong):
- Supabase Realtime uses WAL.
- Hasura Streaming Subscriptions uses an append-only query (could also use WAL).
- Hasura Live Queries uses interval polling, refetching, and multiplexing.
- Supabase uses Postgres RLS for authorization, while Hasura uses an internal RLS system which composes queries (which allows for features like the multiplexing above).
- All 3 use websockets for their client communication.
Supabase Realtime
https://github.com/supabase/realtime#introduction
https://supabase.com/docs/guides/realtime
Hasura Subscriptions
https://github.com/hasura/graphql-engine/blob/master/archite...
https://github.com/hasura/graphql-engine/blob/master/archite...
litestream
-
How (and why) to run SQLite in production
This presentation is focused on the use-case of vertically scaling a single server and driving everything through that app server, which is running SQLite embedded within your application process.
This is the sweet-spot for SQLite applications, but there have been explorations and advances to running SQLite across a network of app servers. LiteFS (https://fly.io/docs/litefs/), the sibling to Litestream for backups (https://litestream.io), is aimed at precisely this use-case. Similarly, Turso (https://turso.tech) is a new-ish managed database company for running SQLite in a more traditional client-server distribution.
-
SQLite3 Replication: A Wizard's Guide🧙🏽
This post intends to help you setup replication for SQLite using Litestream.
wget https://github.com/benbjohnson/litestream/releases/download/v0.3.13/litestream-v0.3.13-linux-amd64.deb sudo dpkg -i litestream-v0.3.13-linux-amd64.deb
-
Pocketbase: Open-source back end in 1 file
Litestream is a library that allows you to easily create backups. You can probably just do analytic queries on the backup data and reduce load on your server.
- Litestream – Disaster recovery and continuous replication for SQLite
-
Why you should probably be using SQLite
One possible strategy is to have one directory/file per customer which is one SQLite file. But then as the user logs in, you have to look up first what database they should be connected to.
OR somehow derive it from the user ID/username. Keeping all the customer databases in a single directory/disk and then constantly "lite streaming" to S3.
Because each user is isolated, they'll be writing to their own database. But migrations would be a pain. They will have to be rolled out to each database separately.
One upside is, you can give users the ability to take their data with them, any time. It is just a single file.
-
Monitor your Websites and Apps using Uptime Kuma
# Builder image FROM docker.io/alpine as BUILDER RUN apk add --no-cache curl jq tar RUN export LITESTREAM_VERSION=$(curl --silent https://api.github.com/repos/benbjohnson/litestream/releases/latest | jq -r .tag_name) && curl -L https://github.com/benbjohnson/litestream/releases/download/${LITESTREAM_VERSION}/litestream-${LITESTREAM_VERSION}-linux-amd64.tar.gz -o litestream.tar.gz && tar xzvf litestream.tar.gz # Main image FROM docker.io/louislam/uptime-kuma as KUMA ARG UPTIME_KUMA_PORT=3001 WORKDIR /app RUN mkdir -p /app/data COPY --from=BUILDER /litestream /usr/local/bin/litestream COPY litestream.yml /etc/litestream.yml COPY run.sh /usr/local/bin/run.sh EXPOSE ${UPTIME_KUMA_PORT} CMD [ "/usr/local/bin/run.sh" ]
Upstream Kuma uses a local SQLite database to store account data, configuration for services to monitor, notification settings, and more. To make sure that our data is available across redeploys, we will bundle Uptime Kuma with Litestream, a project that implements streaming replication for SQLite databases to a remote object storage provider. Effectively, this allows us to treat the local SQLite database as if it were securely stored in a remote database.
-
Backup Grafana SQLite with Litestream using s6-overlay in a container app
FROM docker.io/grafana/grafana-oss:9.5.12-ubuntu # Set USER to root escalating priviliges to perform installation of litestream and s6-overlay USER root RUN apt-get -qq update && \ apt-get -qq install -y xz-utils \ && rm -rf /var/libs/apt/lists/* # https://github.com/benbjohnson/litestream-s6-example/blob/main/Dockerfile # Download the static build of Litestream directly into the path & make it executable. ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.11/litestream-v0.3.11-linux-amd64.tar.gz /tmp/litestream.tar.gz RUN tar -C / -xvzf /tmp/litestream.tar.gz ARG S6_OVERLAY_VERSION="3.1.5.0" # Download the s6-overlay for process supervision. ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz # Copy s6 init & service definitions. COPY etc/s6-overlay /etc/s6-overlay # Copy Litestream configuration file. COPY etc/litestream.yml /etc/litestream.yml # The kill grace time is set to zero because our app handles shutdown through SIGTERM. ENV S6_KILL_GRACETIME=0 # Sync disks is enabled so that data is properly flushed. ENV S6_SYNC_DISKS=1 # Reset USER to 472 to reset the escalated privileges USER 472 # # Run the s6 init process on entry. ENTRYPOINT [ "/init" ]
Litestream is a game changer allowing to backup all your changes to a an s3 compatible cloud storage at 1sec intervals using the WAL all done via the API and not interacting with the DB itself to avoid corruption or impacting performance.
What are some alternatives?
supabase - The open source Firebase alternative.
rqlite - The lightweight, distributed relational database built on SQLite.
pocketbase - Open Source realtime backend in 1 file
debezium - Change data capture for a variety of databases. Please log issues at https://issues.redhat.com/browse/DBZ.
blockscout - Blockchain explorer for Ethereum based network and a tool for inspecting and analyzing EVM based blockchains.
Appwrite - Build like a team of hundreds_
k8s-mediaserver-operator - Repository for k8s Mediaserver Operator project
sqlcipher - SQLCipher is a standalone fork of SQLite that adds 256 bit AES encryption of database files and other security features.
Hasura - Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events.
yugabyte-db - YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
absurd-sql - sqlite3 in ur indexeddb (hopefully a better backend soon)
flyctl - Command line tools for fly.io services