Store SQLite in Cloudflare Durable Objects

• wasi-libc

  48 794 7.7 C

  WASI libc implementation for WebAssembly

  

The wasi-sdk doesn't have support for pthread.h yet (see WebAssembly/wasi-libc#209). A workaround for most libraries is to look at build flags that remove everything related to pthreads from the library you are trying to build.

• do-sqlite

  1 129 3.0 TypeScript

  [Experimental] Persist SQLite in a Cloudflare Durable Object

tl;dr: A custom SQLite virtual file system and some WASM/WASI compilation magic allow to run SQLite on a Cloudflare Worker and persist it into a Durable Object. POC source can be found at github.com/rkusa/do-sqlite.

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• rusqlite

  17 2,737 8.9 Rust

  Ergonomic bindings to SQLite for Rust

  

SQLite is written in C, while workers is based on V8 isolates, so it mainly runs JavaScript. Fortunately, it also supports running WASM through initialising and calling WASM modules via JavaScript. Emscripten can be used to build WASM from C, but I'd rather use it through Rust (using rusqlite), so this is what I focus on right away. Workers can also be written entirely in Rust using worker-rs.

• litestream

  165 9,964 7.5 Go

  Streaming replication for SQLite.

A replication between multiple DO's living on different Cloudflare locations would be neat (maybe based on litestream.io).

• workers-rs

  16 2,262 8.5 Rust

  Write Cloudflare Workers in 100% Rust via WebAssembly

  

SQLite is written in C, while workers is based on V8 isolates, so it mainly runs JavaScript. Fortunately, it also supports running WASM through initialising and calling WASM modules via JavaScript. Emscripten can be used to build WASM from C, but I'd rather use it through Rust (using rusqlite), so this is what I focus on right away. Workers can also be written entirely in Rust using worker-rs.

• rusqlite

  1 2 8.6 Rust

  Ergonomic bindings to SQLite for Rust (by trevyn)

rusqlite = { git = "https://github.com/trevyn/rusqlite.git", branch = "wasm32-unknown-unknown", features = ["bundled"] }

• sqlite-vfs

  2 76 0.0 Rust

  Build SQLite virtual file systems (VFS) by implementing a simple Rust trait.

With that, I had enough information to build sqlite-vfs, a Rust library that wraps all the necessary interactions with the C SQLite library and exposes a much simpler Rust trait. I'll not go into the details of this library here.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• wasm-sqlite

  1 47 0.0 Rust

  [Experimental] SQLite compiled to WASM with pluggable page storage.

The whole code can be found here: github.com/rkusa/wasm-sqlite/blob/main/wasm/src/vfs.rs. By reducing all down to the two methods get_page and put_page (to which I get back later in the post), the VFS ended up being a generic VFS for when you want to persist SQLite pages as separate files/objects. Using it anywhere else than a DO would just be a matter of providing a different implementation to those two methods.

• binaryen

  14 7,099 9.8 WebAssembly

  Optimizer and compiler/toolchain library for WebAssembly

  

Fortunately, the wasm-opt tool has an option to asyncify methods in a WASM. It, therefore, rewrites the WASM and allows to pause and resume the execution. This is exactly what I need to call async functions from within the sync context of the VFS.

• asyncify

  1 116 3.6 JavaScript

  Standalone Asyncify helper for Binaryen

  

The asyncification is achieved by unwinding and rewinding the local stack. With wasm-opt, this is done on a WASM level as post-build transformation. This makes it a language-agnostic approach. Once a WASM module is asyncified it needs to be used through the JavaScript wrapper Asyncify. A good introduction to asyncify can be found here: Using asynchronous web APIs from WebAssembly.

• wasm-bindgen

  44 7,270 9.2 Rust

  Facilitating high-level interactions between Wasm modules and JavaScript

  

While I could theoretically still build for wasm32-unknown-unknown, this target has a major drawback. A lot of libraries automatically use wasm-bindgen as soon as you build for it.

• wasi-sdk

  11 1,131 7.6 Shell

  WASI-enabled WebAssembly C/C++ toolchain

  

The previously mentioned PR for wasm32-unknown-unknown compatibility solved this by including libc .c files from OpenBSD. My go to solution is different though. I prefer to build using the wasi-sdk (a WASI-enabled WebAssembly C/C++ toolchain).

• workers-wasi

  5 119 0.0 C++
  

While there is a WASI implementation for Workers: cloudflare/workers-wasi, I prefer to implement each import manually - especially when there are so few and especially while I am still experimenting. This helps me to keep the full picture of what's going on.

Suggest a related project