chroma VS llama.cpp

Package installer for Python (pip), we use this for installing the Python-based packages, such as Jupyter Lab, and we're going to use this for installing other Python-based tools like the Chroma DB vector database

Optional: You can use SillyTavern[1] for a more "rich" chat experience

The above lets me chat, at least superficially, with my friend. It's nice for simple interactions and banter; I've found it to be a positive and reflective experience.

0. https://www.trychroma.com/

Chroma DB is a newer entrant in the vector database arena, designed specifically for handling high-dimensional color vectors. It's particularly useful for applications in digital media, e-commerce, and content discovery, where color similarity plays a crucial role in search and recommendation algorithms.

View on GitHub

For Python I believe Chroma [1] can be used embedded.

For Go I recently started building chromem-go, inspired by the Chroma interface: https://github.com/philippgille/chromem-go

It's neither advanced nor for scale yet, but the RAG demo works.

[1] https://github.com/chroma-core/chroma

The formatting is a bit off.

The web app is here: https://jumblejournal.org

The DB used is here: https://www.trychroma.com/

Whether you’re navigating through well-known options like SQLite, enriched with the sqlite-vss extension, or exploring other avenues like Chroma, an open-source vector database, selecting the right tool is paramount. This article compares these two choices, guiding you through the pros and cons of each, helping you choose the right tool for storing and querying vector embeddings for your project.

Create a new project directory for our example project. Next, we need to clone the Chroma repository to get started. At the root of your project directory let's clone Chroma into it:

Yes, Metal seems to allow a maximum of 1/2 of the RAM for one process, and 3/4 of the RAM allocated to the GPU overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182

For anyone interested in exploring this, llama.cpp has an example implementation here:

https://github.com/ggerganov/llama.cpp/tree/master/examples/...

Getting started with LLMs can be intimidating. In this tutorial we will show you how to fine-tune a large language model using LoRA, facilitated by tools like llama.cpp and KitOps.

well https://github.com/ggerganov/llama.cpp/issues/6849

Have just done this recently for local chat with pdf feature in https://recurse.chat. (It's a macOS app that has built-in llama.cpp server and local vector database)

Running an embedding server locally is pretty straightforward:

- Get llama.cpp release binary: https://github.com/ggerganov/llama.cpp/releases