go-sqlite VS MeiliSearch

Check out https://github.com/zombiezen/go-sqlite if you're interested in trying out Sqlite in Go again. Nice interface, negligible compile time impact, fast, compiles without CGO. It's very comfortable.

I agree that going from text to sqlite is a bit of a hurdle, especially if you're not writing C :)

struggling figuring out how to make my cgo sqlite cross-compile to Windows

Plenty of people trying to fix that.

There's at least:

https://modernc.org/sqlite

Then there's https://github.com/zombiezen/go-sqlite that actually builds https://crawshaw.io/sqlite on top of modernc.

And there's mine that has both a low level and a database/sql driver builds and runs everywhere Go does: https://github.com/ncruces/go-sqlite3

There are several different Go bindings for SQLite. I maintain https://pkg.go.dev/zombiezen.com/go/sqlite

I like and use https://github.com/zombiezen/go-sqlite for CGo-free SQLite. It avoids some of the problems database/sql has, discussed here: https://crawshaw.io/blog/go-and-sqlite.

> C libraries are required to interact with SQLite

Or: modernc.org/sqlite (https://github.com/zombiezen/go-sqlite), "an automatically generated translation of the original C source code of SQLite into Go"

https://pkg.go.dev/modernc.org/sqlite via https://pkg.go.dev/zombiezen.com/go/sqlite

modernc.org/sqlite provides a database/sql driver, but does not (currently) provide an easy way to get at the more advanced functionality of SQLite, like streaming blob I/O or user-defined functions. David Crawshaw has argued that the database/sql API is not a good fit for SQLite, which is how crawshaw.io/sqlite came about.

We needed to introduce a new service for search. As we settled on using meilisearch, we needed a way to sync updates on our models with the records in meilisearch. We could've continued to use callbacks but we needed something better.

Meilisearch

In this case, a good strategy is to use vector search only when the keyword/prefix search returns none or just a small number of results. A good candidate for this is MeiliSearch. It uses custom ranking rules to provide results as fast as the user can type.

With the VertexAI endpoint set up and tested, our next step is to work with LibreChat. LibreChat is an open-source ChatGPT clone that can integrate with various AI models, including the PaLM 2 models via the VertexAI API. It's built using React, MongoDB, and Meilisearch technologies.

Meilisearch seems like it is the best open source option.

https://www.meilisearch.com/

Starting with v1.3, you can use Meilisearch as a vector store. Meilisearch allows you to store vector embeddings alongside your documents conveniently. You will need to create the vector embeddings using your third-party tool of choice (Hugging Face, OpenAI). As we published the first v1.3 release candidate, you can try out vector search today.

I work at Meilisearch, an open-source search engine built in Rust. 🦀

https://www.meilisearch.com/ https://github.com/meilisearch