trulens VS ML-Workspace

Retrieval using a single vector is called dense passage retrieval (DPR), because an entire passage (dozens to hundreds of tokens) is encoded as a single vector. ColBERT instead encodes a vector-per-token, where each vector is influenced by surrounding context. This leads to meaningfully better results; for example, here’s ColBERT running on Astra DB compared to DPR using openai-v3-small vectors, compared with TruLens for the Braintrust Coda Help Desk data set. ColBERT easily beats DPR at correctness, context relevance, and groundedness.

12. TruLens by TruEra | Github | tutorial

3) Hallucination is probably the biggest problem we solve for. To do evals for hallucination, we typically see our users use a combination of groundedness (does the context support the LLM response) and context relevance (is the retrieved context relevant to the query). There's also a bunch more for the evaluations you mentioned (moderation models, sentiment, usefulness, etc.) and it's pretty easy to add custom evals.

Also - my hot take is that gpt-3.5 is good enough for evals (sometimes better) than gpt-4 if you give the LLM enough instructions on how to do the eval.

website: https://www.trulens.org/

The team at TruEra recently released an open source project for evaluation & tracking of LLM applications called TruLens-Eval. We’ve specifically targeted retrieval-augmented QA as a core use case and so far we’ve seen it used for comparing different models and parameters, prompts, vector-db configurations and query planning strategies. I’d love to get your feedback on it.

Also check out: https://github.com/ml-tooling/ml-workspace, it a nice open source project with lots of packages ready to use.

The other option - no reason why you couldn't host something on the desktop machine - web based IDE like R-Studio or Python - have a look at ml-workspace - https://github.com/ml-tooling/ml-workspace that runs in Docker and would provide interfaces for both Python and R, VSCode as well as a GPU accelerated variant for doing Tensorflow etc - either Windows or Linux can support Docker containers (Linux is less trouble apparently - I only have played with it in Linux personally)

It will be a web based IDE dev kit (like Jupyter Hub, or JupyterLab) if you are familiar with them)

I'll stick with https://github.com/ml-tooling/ml-workspace, is a docker with all tools installed, also the option of using GPU, so I think is better than only for debian. This way anyone can use it.