sqlite-ecosystem VS tg

I definitely plan to! I have a much larger list of SQLite extensions I've built here: https://github.com/asg017/sqlite-ecosystem

Here's a few other references you may enjoy if you wanna learn more about SQLite extensions:

- The single source file for sqlite-vec: https://github.com/asg017/sqlite-vec/blob/main/sqlite-vec.c

- sqlean, a project from Anton Zhiyanov which is good base of great SQLite extensions: https://github.com/nalgeon/sqlean

- The official SQLite docs: https://www.sqlite.org/loadext.html

- The "hello world" SQLite extension example: https://www.sqlite.org/src/file/ext/misc/rot13.c

Author here, happy to answer any questions! My "Introducing sqlite-vss: A SQLite Extension for Vector Search" [0] blog post has more details about the motivations behind this extensions, along with a demo. Includes a semantic search engine hosted on Datasette, build with sqlite-vss and sentence-transformers, on fly.io. Simple and cheap!

Though my favorite part about this extension (and all my other SQLite extensions[1]): you can `pip install sqlite-vss` for Python, `npm install sqlite-vss` for Node.js, or use https://deno.land/x/sqlite_vss for Deno! The underlying SQLite extension has been compiled on different platforms and uploaded to pip/npm/deno.land/x to make distribution easier. There's also pre-compiled extensions on each Github release if you just want to use it with the sqlite3 CLI.

Currently working on getting Mac M1 arm builds, hopefully by end of this week [2]

[0] https://observablehq.com/@asg017/introducing-sqlite-vss

[1] https://github.com/asg017/sqlite-ecosystem

[2] https://github.com/asg017/sqlite-vss/issues/13

Thanks for sharing! I've looked into usearch before, it's really sleek, especially all their language bindings. Though I want sqlite-vec to have total control over what stays in-memory vs on-disk during searches, and most vector search libraries like usearch/hnswlib/faiss/annoy either always store in-memory or don't offer hooks for other storage systems.

Additionally, sqlite-vec takes advantage of some SQLite specific APIs, like BLOB I/O [0], which I hope would speed things up a ton. It's a ton more work, coming up with new storage solutions that are backed by SQLite shadow tables, but I think it'll be work it!

And I also like how sqlite-vec is just a single sqlite-vec.c file. It makes linking + cross-compiling super easy, and since I got burned relying on heavy C++ dependencies with sqlite-vss, a no-dependency rule feels good. Mostly inspired by SQLite single-file sqlite3.c amalgamation, and Josh Baker's single file C projects like tg[1].

[0] https://www.sqlite.org/c3ref/blob_open.html

[1] https://github.com/tidwall/tg

I can't stop precision loss in all cases, but I do my darnedest to avoid loss when it causes false positives, especially for stuff like intersect detection code. For example the collinear [1] function looks really big for a seemingly simple operation, but there are extra checks built in to check for precision loss and in the cases of compiler associate math issues (like a user borking a build with -ffast-math).

I'm sure it's not all perfect but I feel pretty good about it overall. It certainly helps that much of the logic derived from the Tile38 [2] project, which has 8 years of use in production. I ported many of tests too, which makes me warm and fuzzy every time they pass.

[1] https://github.com/tidwall/tg/blob/v0.1.0/tg.c#L389