simsimd VS swiss_army_llama

It’s a good start, but you can’t generally get even remotely close to hardware potential in Rust, let alone Python.

I had to implement a separate C99 library to always trigger the newest SIMD intrinsics, occasionally leveraging SVE on more recent ARM CPUs, that compilers don’t know how to generate.

That library is in turn used in USearch, which is designed for Approximate Search, but some users recently reported that they use it for brute force as well… where it performed 20x faster than FAISS.

https://github.com/unum-cloud/simsimd

Depends what you mean by "local". If you mean in your own home, then there isn't a particularly cheap way unless you have a decent spare machine. If you mean "I get to control everything myself" then you can rent a cheap VPS on a value host like Contabo (you can get 8cores, 30gb of ram, and 1tb SSD on Ubuntu 22.04 for something like $35/month-- just stick the to US data centers).

Then if you want something that is extremely quick and easy to set up and provides a convenient REST api for completions/embeddings with some other nice features, you might want to check out my project here:

https://github.com/Dicklesworthstone/swiss_army_llama

Especially if you use Docker to set it up, you can go from a brand new box to a working setup in under 20 minutes and then access it via the Swagger page from any browser.

I found all of them to be quite bloated and annoying to use directly, which is why I made my own FastAPI based one, Swiss Army Llama. I’m obviously biased, but I far prefer it:

https://github.com/Dicklesworthstone/swiss_army_llama

I just added a very cool feature that lets you supply a sample JSON file and it will automatically generate a BNF grammar for it. You can also supply a pydantic data model description and it will generate the corresponding JSON BNF for you:

https://github.com/Dicklesworthstone/swiss_army_llama/blob/m...

And then you can add that grammar file and it will validate it with this:

https://github.com/Dicklesworthstone/swiss_army_llama/blob/5...

Cool, I also made a similar kind of tool recently that I also shared on HN a couple weeks ago. You might find it useful for generating and managing LLM embeddings locally:

https://github.com/Dicklesworthstone/llama_embeddings_fastap...

Thanks for pointing out those models. I see from a quick Huggingface search that the bge model is available in GGML format. You can trivially add new GGML format models to the code by simply adding the direct download link to this line:

https://github.com/Dicklesworthstone/llama_embeddings_fastap...

So to add the base bge model, you could just add this URL to the list:

https://huggingface.co/maikaarda/bge-base-en-ggml/resolve/ma...

I will add that as an additional default.