TruthfulQA VS JARVIS

Dataset: https://github.com/sylinrl/TruthfulQA

this is a great point.

do you know of any benchmarks doing this today?

given the acute need to evaluate models on contextual factuality, we're exploring how to create a benchmark for this purpose but prefer existing benchmarks if possible.

openai's truthfulqa[0] is close but does not focus on contextual factuality and targets a much harder problem of absolute truth.

if none exist, and people are interested in contributing, please reach out.

[0] https://github.com/sylinrl/TruthfulQA

I agree they show that you can brute-force mimick uncertainty estimates to some degree, and that the model is generally well calibrated (though on what is basically a set of trivia questions, so YMMV)... yet:

Code for https://arxiv.org/abs/2109.07958 found: https://github.com/sylinrl/TruthfulQA

Jarvis: github.com/microsoft/JARVIS

You can build it yourself now. https://github.com/microsoft/JARVIS

https://github.com/microsoft/JARVIS (when ready says >=24GB VRAM)

Can be. There are projects that run fully locally like Microsoft’s Jarvis. https://github.com/microsoft/JARVIS

JARVIS, a system to connect LLMs with ML community (https://github.com/microsoft/JARVIS)

HuggingGPT is a clever idea to boost the capabilities of LLM Agents, and enable them to solve “complicated AI tasks with different domains and modalities”. In short, it uses ChatGPT to plan tasks, select models from Hugging Face (HF), format inputs, execute each subtask via the HF Inference API, and summarise the results. JARVIS tries to generalise this idea, and create a framework to “connect LLMs with the ML community”, which Microsoft Research claims “paves a new way towards advanced artificial intelligence”.