rqlite
litestream
Our great sponsors
rqlite | litestream | |
---|---|---|
112 | 164 | |
14,760 | 9,700 | |
1.6% | - | |
9.9 | 7.5 | |
7 days ago | 7 days ago | |
Go | Go | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
rqlite
-
CursusDB – A new scalable distributed document oriented database
Seems like you could do the same with rqlite [1], since SQLite supports JSON.
[1]: https://rqlite.io
- I'm All-In on Server-Side SQLite
-
So, you want to deploy on the edge?
rqlite[1] creator here, happy to answer any questions. rqlite also supports read-only nodes, which can also help with reads at the "edge". It probably wouldn't scale to 100s of nodes, it is an option.
"rqlite supports adding read-only nodes. You can use this feature to add read scalability to the cluster if you need a high volume of reads, or want to distribute copies of the data nearer to clients – but don’t want those nodes counted towards the quorum. These types of nodes are also known as non-voting nodes."
- LiteFS Cloud: Distributed SQLite with Managed Backups
-
Latest Chat-GPT4 release (May 12th 2023) sure is buggy
result, opErr := db.executeStmtWithConn for the "executeStmt that you , you can manipulate the picture ', you ... i = t given, and it' . If, -212<|endoftext|>
[1] https://rqlite.io/
Yes, it's saved me hours of time -- with certain narrow tasks. For example, it generated these files for me:
https://github.com/rqlite/rqlite/blob/master/auto/backup/sum*
It took a few minutes, might have taken me 1+ hour of some research, and mechanically typing it all in.
-
Ask HN: It's 2023, how do you choose between MySQL and Postgres?
Just to point out, there are now SQLite replication and various "distributed database" projects which seem to work fairly well.
They're probably not as battle tested as the PostgreSQL ones, but they are around, have users, and are actively developed.
The ones I remember off the top of my head:
* https://github.com/rqlite/rqlite <-- more of a "distributed database using RAFT" type of thing
-
rqlite v7.15: the lightweight distributed database built on Go, Raft, and SQLite -- now with automatic backups to S3
Instead of running it as a separate rqlited, integrate it into an existing binary. I answered my own question by looking in https://github.com/rqlite/rqlite/blob/master/cmd/rqlited/main.go
-
Strong Consistency with Raft and SQLite
Is this somehow related to rqlite? https://rqlite.io/
The architecture is very similar.
litestream
-
How (and why) to run SQLite in production
This presentation is focused on the use-case of vertically scaling a single server and driving everything through that app server, which is running SQLite embedded within your application process.
This is the sweet-spot for SQLite applications, but there have been explorations and advances to running SQLite across a network of app servers. LiteFS (https://fly.io/docs/litefs/), the sibling to Litestream for backups (https://litestream.io), is aimed at precisely this use-case. Similarly, Turso (https://turso.tech) is a new-ish managed database company for running SQLite in a more traditional client-server distribution.
-
SQLite3 Replication: A Wizard's Guide🧙🏽
This post intends to help you setup replication for SQLite using Litestream.
wget https://github.com/benbjohnson/litestream/releases/download/v0.3.13/litestream-v0.3.13-linux-amd64.deb sudo dpkg -i litestream-v0.3.13-linux-amd64.deb
-
Pocketbase: Open-source back end in 1 file
Litestream is a library that allows you to easily create backups. You can probably just do analytic queries on the backup data and reduce load on your server.
- Litestream – Disaster recovery and continuous replication for SQLite
-
Why you should probably be using SQLite
One possible strategy is to have one directory/file per customer which is one SQLite file. But then as the user logs in, you have to look up first what database they should be connected to.
OR somehow derive it from the user ID/username. Keeping all the customer databases in a single directory/disk and then constantly "lite streaming" to S3.
Because each user is isolated, they'll be writing to their own database. But migrations would be a pain. They will have to be rolled out to each database separately.
One upside is, you can give users the ability to take their data with them, any time. It is just a single file.
-
Monitor your Websites and Apps using Uptime Kuma
# Builder image FROM docker.io/alpine as BUILDER RUN apk add --no-cache curl jq tar RUN export LITESTREAM_VERSION=$(curl --silent https://api.github.com/repos/benbjohnson/litestream/releases/latest | jq -r .tag_name) && curl -L https://github.com/benbjohnson/litestream/releases/download/${LITESTREAM_VERSION}/litestream-${LITESTREAM_VERSION}-linux-amd64.tar.gz -o litestream.tar.gz && tar xzvf litestream.tar.gz # Main image FROM docker.io/louislam/uptime-kuma as KUMA ARG UPTIME_KUMA_PORT=3001 WORKDIR /app RUN mkdir -p /app/data COPY --from=BUILDER /litestream /usr/local/bin/litestream COPY litestream.yml /etc/litestream.yml COPY run.sh /usr/local/bin/run.sh EXPOSE ${UPTIME_KUMA_PORT} CMD [ "/usr/local/bin/run.sh" ]
Upstream Kuma uses a local SQLite database to store account data, configuration for services to monitor, notification settings, and more. To make sure that our data is available across redeploys, we will bundle Uptime Kuma with Litestream, a project that implements streaming replication for SQLite databases to a remote object storage provider. Effectively, this allows us to treat the local SQLite database as if it were securely stored in a remote database.
-
Backup Grafana SQLite with Litestream using s6-overlay in a container app
FROM docker.io/grafana/grafana-oss:9.5.12-ubuntu # Set USER to root escalating priviliges to perform installation of litestream and s6-overlay USER root RUN apt-get -qq update && \ apt-get -qq install -y xz-utils \ && rm -rf /var/libs/apt/lists/* # https://github.com/benbjohnson/litestream-s6-example/blob/main/Dockerfile # Download the static build of Litestream directly into the path & make it executable. ADD https://github.com/benbjohnson/litestream/releases/download/v0.3.11/litestream-v0.3.11-linux-amd64.tar.gz /tmp/litestream.tar.gz RUN tar -C / -xvzf /tmp/litestream.tar.gz ARG S6_OVERLAY_VERSION="3.1.5.0" # Download the s6-overlay for process supervision. ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-noarch.tar.xz /tmp RUN tar -C / -Jxpf /tmp/s6-overlay-noarch.tar.xz ADD https://github.com/just-containers/s6-overlay/releases/download/v${S6_OVERLAY_VERSION}/s6-overlay-x86_64.tar.xz /tmp RUN tar -C / -Jxpf /tmp/s6-overlay-x86_64.tar.xz # Copy s6 init & service definitions. COPY etc/s6-overlay /etc/s6-overlay # Copy Litestream configuration file. COPY etc/litestream.yml /etc/litestream.yml # The kill grace time is set to zero because our app handles shutdown through SIGTERM. ENV S6_KILL_GRACETIME=0 # Sync disks is enabled so that data is properly flushed. ENV S6_SYNC_DISKS=1 # Reset USER to 472 to reset the escalated privileges USER 472 # # Run the s6 init process on entry. ENTRYPOINT [ "/init" ]
Litestream is a game changer allowing to backup all your changes to a an s3 compatible cloud storage at 1sec intervals using the WAL all done via the API and not interacting with the DB itself to avoid corruption or impacting performance.
What are some alternatives?
dqlite - Embeddable, replicated and fault-tolerant SQL engine.
pocketbase - Open Source realtime backend in 1 file
cockroach - CockroachDB - the open source, cloud-native distributed SQL database.
realtime - Broadcast, Presence, and Postgres Changes via WebSockets
bolt
etcd - Distributed reliable key-value store for the most critical data of a distributed system [Moved to: https://github.com/etcd-io/etcd]
TinyGo - Go compiler for small places. Microcontrollers, WebAssembly (WASM/WASI), and command-line tools. Based on LLVM.
go-cache - An in-memory key:value store/cache (similar to Memcached) library for Go, suitable for single-machine applications.
k8s-mediaserver-operator - Repository for k8s Mediaserver Operator project
groupcache - groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.
budibase - Budibase is an open-source low code platform that helps you build internal tools in minutes 🚀
sqlcipher - SQLCipher is a standalone fork of SQLite that adds 256 bit AES encryption of database files and other security features.