[R] LMFlow Benchmark: An Automatic Evaluation Framework for Open-Source LLMs

• lm-evaluation-harness

  34 5,211 9.9 Python

  A framework for few-shot evaluation of language models.

  

Here, we make use of Eleuther AI’s LM evaluation harness repository (https://github.com/EleutherAI/lm-evaluation-harness) to get QA accuracy results. We also evaluate all models’ NLL metrics on their datasets, with their questions as contexts and answers as output sentences.

• LMFlow

  10 8,042 9.6 Python

  An Extensible Toolkit for Finetuning and Inference of Large Foundation Models. Large Models for All.

  

LMFlow: https://github.com/OptimalScale/LMFlow

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• giskard

  7 3,164 10.0 Python

  🐢 Open-Source Evaluation & Testing framework for LLMs and ML models

  

This is super interesting! Thanks for sharing. We're also working on this research field from an open-source angle (https://github.com/Giskard-AI/giskard)

Suggest a related project