quivr VS trulens

3. Quivr | GitHub | tutorial

I used Ollama (with Mistral 7B) and Quivr to get a local RAG up and running and it works fine, but was surprised to find there are no easy user-friendly ways to do it. Most other local LLM UIs don't implement this use case (I looked here), even though it is one of the most useful local LLM use-cases I can think of: search and summarize information from sensitive / confidential documents.

In my opinion LLM based document search tools such as OSS Quivr may be better suited for documentation search for startups.

A highly customed Quivr with one of the 'Open Source LLMs' may provides great 'semantic search' for product documentation.

https://github.com/StanGirard/quivr

Hey thanks for creating this, will try later if i have time. Meanwhile, do you try some of other second brain app such as this, and how was the comparison? The one i mentioned was trending on github so i think its decent (been playing with it since last week or so, also). But i already starred your repo so i can come back later.

Retrieval using a single vector is called dense passage retrieval (DPR), because an entire passage (dozens to hundreds of tokens) is encoded as a single vector. ColBERT instead encodes a vector-per-token, where each vector is influenced by surrounding context. This leads to meaningfully better results; for example, here’s ColBERT running on Astra DB compared to DPR using openai-v3-small vectors, compared with TruLens for the Braintrust Coda Help Desk data set. ColBERT easily beats DPR at correctness, context relevance, and groundedness.

12. TruLens by TruEra | Github | tutorial

3) Hallucination is probably the biggest problem we solve for. To do evals for hallucination, we typically see our users use a combination of groundedness (does the context support the LLM response) and context relevance (is the retrieved context relevant to the query). There's also a bunch more for the evaluations you mentioned (moderation models, sentiment, usefulness, etc.) and it's pretty easy to add custom evals.

Also - my hot take is that gpt-3.5 is good enough for evals (sometimes better) than gpt-4 if you give the LLM enough instructions on how to do the eval.

website: https://www.trulens.org/

The team at TruEra recently released an open source project for evaluation & tracking of LLM applications called TruLens-Eval. We’ve specifically targeted retrieval-augmented QA as a core use case and so far we’ve seen it used for comparing different models and parameters, prompts, vector-db configurations and query planning strategies. I’d love to get your feedback on it.