SQLite Scales Read Concurrency

• HeapProfiler

  1 57 10.0 C#

  Windows Heap Profiler Frontend

The app was a C# frontend (https://github.com/kg/HeapProfiler) that drove windows OS services to take heap snapshots and capture stack traces, so I ended up writing a custom key/value store in C# to avoid having to do cross-language interop, marshaling, etc (the cost of sending blobs to SQLite and running queries was adding up.). It's hard to beat the best-in-class optimized databases on their own turf but if you can just grab a spot to dump your data into, you end up being a lot faster.

By the end it ran fast enough that it was able to saturate the kernel's paging infrastructure and make my mouse cursor stutter, and I was able to take 1-2 snapshots per second of a full running Firefox process with real webpages in it, so it was satisfactory. SQLite couldn't process the amount of data I was pumping in at that rate (but it still performed pretty well - maybe a few snapshots per minute)

• sql3

  1 5 0.0 JavaScript

  Wrap better-sqlite3 to provide fast writes in cluster mode, migrations, and query builders.

Using cluster and worker threads. The code is here, but completely undocumented, as I’ve been side tracked by a few other things[0]. I’m currently looking into porting that project to Bun, so it’s possibly dead in its current form.

I was planning on turning that into a library, but Bun nerd sniped me.

You need the “hey” tool to run the benchmarks[1] the way that I was running them.

[0] https://github.com/chrisdavies/sql3

[1] https://github.com/rakyll/hey

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• Hey

  38 17,249 0.0 Go

  HTTP load generator, ApacheBench (ab) replacement (by rakyll)

Using cluster and worker threads. The code is here, but completely undocumented, as I’ve been side tracked by a few other things[0]. I’m currently looking into porting that project to Bun, so it’s possibly dead in its current form.

I was planning on turning that into a library, but Bun nerd sniped me.

You need the “hey” tool to run the benchmarks[1] the way that I was running them.

[0] https://github.com/chrisdavies/sql3

[1] https://github.com/rakyll/hey

Suggest a related project