torcheval VS ollama

Reading this could make people believe it is computed from the probability distribution of the model alone.

To be clearer, it is the exponent of the average negative log probability that the model gives to the real tokens of a sample text[0]. Roughly, it relates to how strongly the model can predict the sample text. A perfect model would have zero perplexity; a random model has a perplexity equal to the number of possible tokens; the worst model has infinite perplexity.

[0]: https://github.com/pytorch/torcheval/blob/3faf19c060b8a7c074...

Shameless plug, my team works on torcheval and torchtnt. Neither of them are core pytorch, but if you're looking to help build out tooling for metric evaluation or training frameworks, both libraries are pretty new with very low hanging fruit.

Hey I work on TorchEval let us know if we can be of any help here :)

To support the exploration, I've developed a simple Retrieval Augmented Generation (RAG) workflow that works completely locally on the laptop for free. If you're interested, you can find the code itself here. Basically, I've used Testcontainers to create a Postgres database container with the pgvector extension to store text embeddings and an open source LLM with which I send requests to: Meta's llama3 through ollama.

Note: Before proceeding further you need to download and run Ollama, you can do so by clicking here.

`file://*` URLs are now allowed => ollama works with simple html files now

https://github.com/ollama/ollama/commit/1a29e9a879433fc55cf1...

This guide assumes you have a supported NVIDIA GPU and have installed Ubuntu 22.04 on the machine that will host the ollama docker image. AMD is now supported with ollama but this guide does not cover this type of setup.

Nowadays, running powerful LLMs locally is ridiculously easy when using tools such as ollama. Just follow the installation instructions for your #OS. From now on, we'll assume using bash on Ubuntu.

I focused on supporting Ollama, GPT4All, and LMStudio, all of which run smoothly on a Mac computer. Many of these tools are user-friendly wrappers around Llama.cpp, allowing easy model downloads and providing a REST interface to query the available models. Last week, I also added "👋🏼 Jan" support because HuggingFace has endorsed this provider out-of-the-box.