quinn
litestream
Our great sponsors
quinn | litestream | |
---|---|---|
23 | 164 | |
3,396 | 9,700 | |
3.7% | - | |
9.0 | 7.5 | |
9 days ago | 6 days ago | |
Rust | Go | |
Apache License 2.0 | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
quinn
-
Why HTTP/3 is eating the world
Since it lives on top of UDP, I believe all you need is SOCK_DGRAM, right? The rest of QUIC can be in a userspace library ergonomically designed for your programming language e.g. https://github.com/quinn-rs/quinn - and can interoperate with others who have made different choices.
Alternately, if you need even higher performance, DPDK gives the abstractions you'd need; see e.g. https://dl.acm.org/doi/abs/10.1145/3565477.3569154 on performance characteristics.
-
Async rust – are we doing it all wrong?
> Making things thread safe for runtime-agnostic utilities like WebSocket is yet another price we pay for making everything multi-threaded by default. The standard way of doing what I'm doing in my code above would be to spawn one of the loops on a separate background task, which could land on a separate thread, meaning we must do all that synchronization to manage reading and writing to a socket from different threads for no good reason.
Why so? Libraries like quinn[1] define "no IO" crate to define runtime-agnostic protocol implementation. In this way we won't suffer by forcing ourselves using synchronization primitives.
Also, IMO it's relatively easy to use Send-bounded future in non-Send(i.o.w. single-threaded) runtime environment, but it's almost impossible to do opposite. Ecosystem users can freely use single threaded async runtime, but ecosystem providers should not. If you want every users to only use single threaded runtime, it's a major loss for the Rust ecosystem.
Typechecked Send/Sync bounds are one of the holy grails that Rust provides. Albeit it's overkill to use multithreaded async runtimes for most users, we should not abandon them because it opens an opportunity for high-end users who might seek Rust for their high-performance backends.
-
quicssh-rs Rust implementation SSH over Quic proxy tool
quicssh-rs is quicssh rust implementation. It is based on quinn and tokio
-
The birth of a package manager [written in Rust :)]
Regarding Quinn, I had a blast this week resurrecting an old PR. Looking forward to the next!
- Best performing quic implementation?
-
str0m a sans I/O WebRTC library
By studying u/djcu/hachyderm.io (and others!) excellent work in Quinn, doing a sans I/O implementation of QUIC https://github.com/quinn-rs/quinn we have a way forward.
-
durian - a high-level general purpose client/server networking library
QUIC isn't web/wasm-compatible because of https://github.com/quinn-rs/quinn/issues/1388, so durian wouldn't either since it's built on top of it.
-
FPS server with QUINN?
Quinn, as in the implementation of QUIC? https://github.com/quinn-rs/quinn
-
I built a Zoom clone 100% IN RUST
You are right, I am planning to switch the transport to UDP + quic using the awesome QUINN library, https://github.com/quinn-rs/quinn .
-
I write a secure UDP tunnel
Hi, I am new to the community, I just started learning rust and created a secure UDP tunnel based on the Quinn library, thanks to Quinn, I didn't need to go into the detail of the QUIC protocol and quickly created a UDP tunnel, and thanks to the BBR congestion control algorithm it uses, the tunnel performs quite well with lousy and long fat network, I didn't do any benchmark, but it performs a lot better (higher throughput with LFN) than most of other TCP tunnel implementations I used before.
litestream
-
How (and why) to run SQLite in production
This presentation is focused on the use-case of vertically scaling a single server and driving everything through that app server, which is running SQLite embedded within your application process.
This is the sweet-spot for SQLite applications, but there have been explorations and advances to running SQLite across a network of app servers. LiteFS (https://fly.io/docs/litefs/), the sibling to Litestream for backups (https://litestream.io), is aimed at precisely this use-case. Similarly, Turso (https://turso.tech) is a new-ish managed database company for running SQLite in a more traditional client-server distribution.
-
SQLite3 Replication: A Wizard's Guide🧙🏽
This post intends to help you setup replication for SQLite using Litestream.
wget https://github.com/benbjohnson/litestream/releases/download/v0.3.13/litestream-v0.3.13-linux-amd64.deb sudo dpkg -i litestream-v0.3.13-linux-amd64.deb
-
Pocketbase: Open-source back end in 1 file
Litestream is a library that allows you to easily create backups. You can probably just do analytic queries on the backup data and reduce load on your server.
- Litestream – Disaster recovery and continuous replication for SQLite
-
Why you should probably be using SQLite
One possible strategy is to have one directory/file per customer which is one SQLite file. But then as the user logs in, you have to look up first what database they should be connected to.
OR somehow derive it from the user ID/username. Keeping all the customer databases in a single directory/disk and then constantly "lite streaming" to S3.
Because each user is isolated, they'll be writing to their own database. But migrations would be a pain. They will have to be rolled out to each database separately.
One upside is, you can give users the ability to take their data with them, any time. It is just a single file.
-
Monitor your Websites and Apps using Uptime Kuma
# Builder image FROM docker.io/alpine as BUILDER RUN apk add --no-cache curl jq tar RUN export LITESTREAM_VERSION=$(curl --silent https://api.github.com/repos/benbjohnson/litestream/releases/latest | jq -r .tag_name) && curl -L https://github.com/benbjohnson/litestream/releases/download/${LITESTREAM_VERSION}/litestream-${LITESTREAM_VERSION}-linux-amd64.tar.gz -o litestream.tar.gz && tar xzvf litestream.tar.gz # Main image FROM docker.io/louislam/uptime-kuma as KUMA ARG UPTIME_KUMA_PORT=3001 WORKDIR /app RUN mkdir -p /app/data COPY --from=BUILDER /litestream /usr/local/bin/litestream COPY litestream.yml /etc/litestream.yml COPY run.sh /usr/local/bin/run.sh EXPOSE ${UPTIME_KUMA_PORT} CMD [ "/usr/local/bin/run.sh" ]
Upstream Kuma uses a local SQLite database to store account data, configuration for services to monitor, notification settings, and more. To make sure that our data is available across redeploys, we will bundle Uptime Kuma with Litestream, a project that implements streaming replication for SQLite databases to a remote object storage provider. Effectively, this allows us to treat the local SQLite database as if it were securely stored in a remote database.
-
Backup Grafana SQLite with Litestream using s6-overlay in a container app
FROM docker.io/grafana/grafana-oss:9.5.12-ubuntu # Set USER to root escalating priviliges to perform installation of litestream and s6-overlay USER root RUN apt-get -qq update && \ apt-get -qq install -y xz-utils \ && rm -rf /var/libs/apt/lists/* # https://github.com/benbjohnson/litestream-s6-example/blob/main/Dockerfile # Download the static build of Litestream directly into the path & make it executable. ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.11/litestream-v0.3.11-linux-amd64.tar.gz /tmp/litestream.tar.gz RUN tar -C / -xvzf /tmp/litestream.tar.gz ARG S6_OVERLAY_VERSION="3.1.5.0" # Download the s6-overlay for process supervision. ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz # Copy s6 init & service definitions. COPY etc/s6-overlay /etc/s6-overlay # Copy Litestream configuration file. COPY etc/litestream.yml /etc/litestream.yml # The kill grace time is set to zero because our app handles shutdown through SIGTERM. ENV S6_KILL_GRACETIME=0 # Sync disks is enabled so that data is properly flushed. ENV S6_SYNC_DISKS=1 # Reset USER to 472 to reset the escalated privileges USER 472 # # Run the s6 init process on entry. ENTRYPOINT [ "/init" ]
Litestream is a game changer allowing to backup all your changes to a an s3 compatible cloud storage at 1sec intervals using the WAL all done via the API and not interacting with the DB itself to avoid corruption or impacting performance.
What are some alternatives?
quiche - 🥧 Savoury implementation of the QUIC transport protocol and HTTP/3
s2n-quic - An implementation of the IETF QUIC protocol
rqlite - The lightweight, distributed relational database built on SQLite.
pocketbase - Open Source realtime backend in 1 file
h3
realtime - Broadcast, Presence, and Postgres Changes via WebSockets
msquic - Cross-platform, C implementation of the IETF QUIC protocol, exposed to C, C++, C# and Rust.
k8s-mediaserver-operator - Repository for k8s Mediaserver Operator project
sqlcipher - SQLCipher is a standalone fork of SQLite that adds 256 bit AES encryption of database files and other security features.
flyctl - Command line tools for fly.io services
litefs - FUSE-based file system for replicating SQLite databases across a cluster of machines
datasette - An open source multi-tool for exploring and publishing data