datasette.io
dqlite
datasette.io | dqlite | |
---|---|---|
6 | 33 | |
81 | 3,725 | |
- | 1.1% | |
8.0 | 9.5 | |
24 days ago | 18 days ago | |
HTML | C | |
- | GNU General Public License v3.0 or later |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
datasette.io
-
Architecture Notes: Datasette
Opened an issue exploring alternatives here: https://github.com/simonw/datasette.io/issues/109
I decided to just drop "any size" but keep "any shape".
-
How to have git pushes auto-deploy to a remote server?
Here's an example from one of my projects: https://github.com/simonw/datasette.io/blob/main/.github/workflows/deploy.yml
-
Schema on write is better to live by
I've come around to almost the opposite approach.
I pull all of the data I can get my hands on (from Twitter, GitHub, Swarm, Apple Health, Pocket, Apple Photos and more) into SQLite database tables that match the schema of the system that they are imported from.
For my own personal Dogsheep (https://simonwillison.net/2020/Nov/14/personal-data-warehous...) that's 119 tables right now.
Then I use SQL queries against those tables to extract and combine data in ways that are useful to me.
If the schema of the systems I am importing from changes, I can update my queries to compensate for the change.
This protects me from having to solve for a standard schema up front - I take whatever those systems give me. But it lets me combine and search across all of the data from disparate systems essentially at runtime.
I even have a search engine for this, which is populated by SQL queries against the different source tables. You can see an example of how that works at https://github.com/simonw/datasette.io/blob/main/templates/d... - which powers the search interface at https://datasette.io/-/beta
-
Using sqlite3 as a notekeeping document graph
I've been exploring this technique more over the past year and I really like it - https://datasette.io (code at https://github.com/simonw/datasette.io ) is a more recent and much more complicated example.
Extracting links from markdown and using them to populate some additional columns or tables at build time would be pretty straight forward.
- Ask HN: What novel tools are you using to write web sites/apps?
-
What's New in SQLite 3.35
I run SQLite in serverless environments (Cloud Run, Vercel, Heroku) for dozens of projects... but the trick is that they all treat the database as a read-only asset.
If I want to deploy updated data, I build a brand new image and deploy the application bundled with the data. I tend to run the deploys for these (including the database build) in GitHub Actions workflows.
This works really well, but only for applications that don't need to apply constant updates more than a few times an hour! If you have a constant stream of updates I still think you're better off using a hosted database like Heroku PostgreSQL or Google Cloud SQL.
One example of a site I deploy like that is https://datasette.io/ - it's built and deployed by this GitHub Actions workflow here: https://github.com/simonw/datasette.io/blob/main/.github/wor...
dqlite
-
Marmot: Multi-writer distributed SQLite based on NATS
If you're interested in this, here are some related projects that all take slightly different approaches:
- LiteSync directly competes with Marmot and supports DDL sync, but is closed source commercial (similar to SQLite EE): https://litesync.io
- dqlite is Canonical's distributed SQLite that depends on c-raft and kernel-level async I/O: https://dqlite.io
- cr-sqlite is a Rust-based loadable extension that adds CRDT changeset generation and reconciliation to SQLite: https://github.com/vlcn-io/cr-sqlite
Slightly related but not really (no multi writer, no C-level SQLite API or other restrictions):
- comdb2 (Bloombergs multi-homed RDMS using SQLite as the frontend)
- rqlite: RDMS with HTTP API and SQLite as the storage engine, used for replication and strong consistency (does not scale writes)
- litestream/LiteFS: disaster recovery replication
- liteserver: active read-only replication (predecessor of LiteSync)
- I'm All-In on Server-Side SQLite
-
SQLite performance tuning: concurrent reads, multiple GBs and 100k SELECTs/s
I'd be curious for a similar tuning with Dqlite: https://github.com/canonical/dqlite
- Strong Consistency with Raft and SQLite
-
9 years of open-source database development: reviewing the designs
Anyone knows how the DB this is about, https://rqlite.io/, compares with https://dqlite.io/ by Canonical (both seem to be distributed versions of sqlite)?
- SQLite the only database you will ever need in most cases
-
Transcending Posix: The End of an Era?
For folks' context, the new tool that's being discussed in the thread mentioned by the parent here is litefs [0], as well as which you can also look at rqlite [1] and dqlite [2], which all provide different trade-offs (e.g. rqlite is 'more strongly consistent' than litefs).
[0]: https://github.com/superfly/litefs
[1]: https://github.com/rqlite/rqlite
[2]: https://github.com/canonical/dqlite
-
SQLite is not a toy database
I presume you're familiar with https://github.com/canonical/dqlite (made by my employer) and https://github.com/rqlite/rqlite (unrelated)? How will mvsqlite compare to those?
-
GitDB, a distributed embeddable database on top of Git
Check out dqlite, it's sqlite but with a raft consensus to distribute changes through a log: https://dqlite.io/ You can link it in as a library too, it sounds like exactly what you want.
- Ask HN: Free and open source distributed database written in C++ or C
What are some alternatives?
datasette - An open source multi-tool for exploring and publishing data
rqlite - The lightweight, distributed relational database built on SQLite.
gomodest - A complex SAAS starter kit using Go, the html/template package, and sprinkles of javascript.
kine - Run Kubernetes on MySQL, Postgres, sqlite, dqlite, not etcd.
org-roam-server - A Web Application to Visualize the Org-Roam Database
litestream - Streaming replication for SQLite.
openapi-generator - OpenAPI Generator allows generation of API client libraries (SDK generation), server stubs, documentation and configuration automatically given an OpenAPI Spec (v2, v3)
better-sqlite3 - The fastest and simplest library for SQLite3 in Node.js.
headlessui - Completely unstyled, fully accessible UI components, designed to integrate beautifully with Tailwind CSS.
boringproxy - Simple tunneling reverse proxy with a fast web UI and auto HTTPS. Designed for self-hosters.
SvelteKit - web development, streamlined
Bedrock - Rock solid distributed database specializing in active/active automatic failover and WAN replication