Store SQLite in Cloudflare Durable Objects

This page summarizes the projects mentioned and recommended in the original post on dev.to

InfluxDB - Power Real-Time Data Analytics at Scale
Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
www.influxdata.com
featured
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com
featured
  • wasi-libc

    WASI libc implementation for WebAssembly

  • The wasi-sdk doesn't have support for pthread.h yet (see WebAssembly/wasi-libc#209). A workaround for most libraries is to look at build flags that remove everything related to pthreads from the library you are trying to build.

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

    InfluxDB logo
  • do-sqlite

    [Experimental] Persist SQLite in a Cloudflare Durable Object

  • tl;dr: A custom SQLite virtual file system and some WASM/WASI compilation magic allow to run SQLite on a Cloudflare Worker and persist it into a Durable Object. POC source can be found at github.com/rkusa/do-sqlite.

  • rusqlite

    Ergonomic bindings to SQLite for Rust

  • SQLite is written in C, while workers is based on V8 isolates, so it mainly runs JavaScript. Fortunately, it also supports running WASM through initialising and calling WASM modules via JavaScript. Emscripten can be used to build WASM from C, but I'd rather use it through Rust (using rusqlite), so this is what I focus on right away. Workers can also be written entirely in Rust using worker-rs.

  • litestream

    Streaming replication for SQLite.

  • A replication between multiple DO's living on different Cloudflare locations would be neat (maybe based on litestream.io).

  • workers-rs

    Write Cloudflare Workers in 100% Rust via WebAssembly

  • SQLite is written in C, while workers is based on V8 isolates, so it mainly runs JavaScript. Fortunately, it also supports running WASM through initialising and calling WASM modules via JavaScript. Emscripten can be used to build WASM from C, but I'd rather use it through Rust (using rusqlite), so this is what I focus on right away. Workers can also be written entirely in Rust using worker-rs.

  • rusqlite

    Ergonomic bindings to SQLite for Rust (by trevyn)

  • rusqlite = { git = "https://github.com/trevyn/rusqlite.git", branch = "wasm32-unknown-unknown", features = ["bundled"] }

  • sqlite-vfs

    Build SQLite virtual file systems (VFS) by implementing a simple Rust trait.

  • With that, I had enough information to build sqlite-vfs, a Rust library that wraps all the necessary interactions with the C SQLite library and exposes a much simpler Rust trait. I'll not go into the details of this library here.

  • SaaSHub

    SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

    SaaSHub logo
  • wasm-sqlite

    [Experimental] SQLite compiled to WASM with pluggable page storage.

  • The whole code can be found here: github.com/rkusa/wasm-sqlite/blob/main/wasm/src/vfs.rs. By reducing all down to the two methods get_page and put_page (to which I get back later in the post), the VFS ended up being a generic VFS for when you want to persist SQLite pages as separate files/objects. Using it anywhere else than a DO would just be a matter of providing a different implementation to those two methods.

  • binaryen

    Optimizer and compiler/toolchain library for WebAssembly

  • Fortunately, the wasm-opt tool has an option to asyncify methods in a WASM. It, therefore, rewrites the WASM and allows to pause and resume the execution. This is exactly what I need to call async functions from within the sync context of the VFS.

  • asyncify

    Standalone Asyncify helper for Binaryen

  • The asyncification is achieved by unwinding and rewinding the local stack. With wasm-opt, this is done on a WASM level as post-build transformation. This makes it a language-agnostic approach. Once a WASM module is asyncified it needs to be used through the JavaScript wrapper Asyncify. A good introduction to asyncify can be found here: Using asynchronous web APIs from WebAssembly.

  • wasm-bindgen

    Facilitating high-level interactions between Wasm modules and JavaScript

  • While I could theoretically still build for wasm32-unknown-unknown, this target has a major drawback. A lot of libraries automatically use wasm-bindgen as soon as you build for it.

  • wasi-sdk

    WASI-enabled WebAssembly C/C++ toolchain

  • The previously mentioned PR for wasm32-unknown-unknown compatibility solved this by including libc .c files from OpenBSD. My go to solution is different though. I prefer to build using the wasi-sdk (a WASI-enabled WebAssembly C/C++ toolchain).

  • workers-wasi

  • While there is a WASI implementation for Workers: cloudflare/workers-wasi, I prefer to implement each import manually - especially when there are so few and especially while I am still experimenting. This helps me to keep the full picture of what's going on.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts

  • WebAssembly: A promising technology that is quietly being enshitified

    1 project | news.ycombinator.com | 22 May 2024
  • Time-series data ingestion from Rust WebAssembly application, leveraging GreptimeDB and WasmEdge

    2 projects | dev.to | 22 Dec 2023
  • Show HN: Pypipe – A Python command-line tool for pipeline processing

    6 projects | news.ycombinator.com | 22 Oct 2023
  • Wasmer – Run, Publish and Deploy any code, anywhere

    1 project | /r/hypeurls | 19 Sep 2023
  • Support Wasix in Zig Lang

    1 project | news.ycombinator.com | 16 Sep 2023