Store SQLite in Cloudflare Durable Objects

• wasi-libc

  22 648 6.6 C

  WASI libc implementation for WebAssembly

  

  The wasi-sdk doesn't have support for pthread.h yet (see WebAssembly/wasi-libc#209). A workaround for most libraries is to look at build flags that remove everything related to pthreads from the library you are trying to build.

• do-sqlite

  0 116 0.0 TypeScript

  [Experimental] Persist SQLite in a Cloudflare Durable Object

  tl;dr: A custom SQLite virtual file system and some WASM/WASI compilation magic allow to run SQLite on a Cloudflare Worker and persist it into a Durable Object. POC source can be found at github.com/rkusa/do-sqlite.

• SonarLint

  www.sonarlint.org

  sponsored

  Clean code begins in your IDE with SonarLint. Up your coding game and discover issues early. SonarLint is a free plugin that helps you find & fix bugs and security issues from the moment you start writing code. Install from your favorite IDE marketplace today.

  

• rusqlite

  5 1,997 7.1 Rust

  Ergonomic bindings to SQLite for Rust

  

  SQLite is written in C, while workers is based on V8 isolates, so it mainly runs JavaScript. Fortunately, it also supports running WASM through initialising and calling WASM modules via JavaScript. Emscripten can be used to build WASM from C, but I'd rather use it through Rust (using rusqlite), so this is what I focus on right away. Workers can also be written entirely in Rust using worker-rs.

• litestream

  63 8,277 6.2 Go

  Streaming replication for SQLite.

  A replication between multiple DO's living on different Cloudflare locations would be neat (maybe based on litestream.io).

• workers-rs

  9 1,536 7.7 Rust

  Write Cloudflare Workers in 100% Rust via WebAssembly

  

  SQLite is written in C, while workers is based on V8 isolates, so it mainly runs JavaScript. Fortunately, it also supports running WASM through initialising and calling WASM modules via JavaScript. Emscripten can be used to build WASM from C, but I'd rather use it through Rust (using rusqlite), so this is what I focus on right away. Workers can also be written entirely in Rust using worker-rs.

• rusqlite

  0 0 0.0 Rust

  Ergonomic bindings to SQLite for Rust (by trevyn)

  rusqlite = { git = "https://github.com/trevyn/rusqlite.git", branch = "wasm32-unknown-unknown", features = ["bundled"] }

• sqlite-vfs

  1 57 5.0 Rust

  Build SQLite virtual file systems (VFS) by implementing a simple Rust trait.

  With that, I had enough information to build sqlite-vfs, a Rust library that wraps all the necessary interactions with the C SQLite library and exposes a much simpler Rust trait. I'll not go into the details of this library here.

• InfluxDB

  www.influxdata.com

  sponsored

  Access the most powerful time series database as a service. Ingest, store, & analyze all types of time series data in a fully-managed, purpose-built database. Keep data forever with low-cost storage and superior data compression.

  

• wasm-sqlite

  0 39 3.5 Rust

  [Experimental] SQLite compiled to WASM with pluggable page storage.

  The whole code can be found here: github.com/rkusa/wasm-sqlite/blob/main/wasm/src/vfs.rs. By reducing all down to the two methods get_page and put_page (to which I get back later in the post), the VFS ended up being a generic VFS for when you want to persist SQLite pages as separate files/objects. Using it anywhere else than a DO would just be a matter of providing a different implementation to those two methods.

• binaryen

  7 6,285 9.3 WebAssembly

  Optimizer and compiler/toolchain library for WebAssembly

  

  Fortunately, the wasm-opt tool has an option to asyncify methods in a WASM. It, therefore, rewrites the WASM and allows to pause and resume the execution. This is exactly what I need to call async functions from within the sync context of the VFS.

• asyncify

  0 95 0.0 JavaScript

  Standalone Asyncify helper for Binaryen

  

  The asyncification is achieved by unwinding and rewinding the local stack. With wasm-opt, this is done on a WASM level as post-build transformation. This makes it a language-agnostic approach. Once a WASM module is asyncified it needs to be used through the JavaScript wrapper Asyncify. A good introduction to asyncify can be found here: Using asynchronous web APIs from WebAssembly.

• wasm-bindgen

  14 6,187 8.2 Rust

  Facilitating high-level interactions between Wasm modules and JavaScript

  

  While I could theoretically still build for wasm32-unknown-unknown, this target has a major drawback. A lot of libraries automatically use wasm-bindgen as soon as you build for it.

• wasi-sdk

  6 841 5.4 Shell

  WASI-enabled WebAssembly C/C++ toolchain

  

  The previously mentioned PR for wasm32-unknown-unknown compatibility solved this by including libc .c files from OpenBSD. My go to solution is different though. I prefer to build using the wasi-sdk (a WASI-enabled WebAssembly C/C++ toolchain).

• workers-wasi

  2 84 0.0 C++

  

  While there is a WASI implementation for Workers: cloudflare/workers-wasi, I prefer to implement each import manually - especially when there are so few and especially while I am still experimenting. This helps me to keep the full picture of what's going on.

Suggest a related project