perses VS promptbench

Perses is also the name of this Prometheus dataviz project: https://github.com/perses/perses

Funny that the God of Destruction is such a hot name lately

You could take a look at Perses https://github.com/perses/perses

Grafana seems to be the most popular so far. But as it develops, it becomes more and more bulky and complex.

https://github.com/perses/perses seems to be a very promising one.

Fair question.

Evaluate refers to the phase after training to check if the training is good.

Usually the flow goes training -> evaluation -> deployment (what you called inference). This project is aimed for evaluation. Evaluation can be slow (might even be slower than training if you're finetuning on a small domain specific subset)!

So there are [quite](https://github.com/microsoft/promptbench) [a](https://github.com/confident-ai/deepeval) [few](https://github.com/openai/evals) [frameworks](https://github.com/EleutherAI/lm-evaluation-harness) working on evaluation, however, all of them are quite slow, because LLM are slow if you don't have infinite money. [This](https://github.com/open-compass/opencompass) one tries to speed up by parallelizing on multiple computers, but none of them takes advantage of the fact that many evaluation queries might be similar and all try to evaluate on all given queries. And that's where this project might come in handy.