guardrails VS TruthfulQA

You're absolutely correct, and I agree that there's potentially a risk of quality loss. But likewise, since these are all intrinsically linked, it may be possible to leverage strength by combining these tasks. I'm unaware of a paper reviewing the reliability and/or performance of LLMs in this specific scenario. If you find any, do share :) With regards to generating JSON responses - there are simple ways to nudge the model and even validate it, using libraries such as https://github.com/promptslab/Promptify, https://github.com/eyurtsev/kor and https://github.com/ShreyaR/guardrails

There are a couple different approaches:

- Use multi-shot prompting with something like guardrails to try prompting a commercial model until it works. [1]

- Use a local model with something with a final layer that steers token selection towards syntactically valid tokens [2]

[1] https://github.com/ShreyaR/guardrails

[2] "Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema" @ https://github.com/newhouseb/clownfish.

👍 validate and correct the outputs of LLMs using guardrails

not vouching for it, but I know this is at least a thing that exists and I like the general idea: https://github.com/shreyar/guardrails

Dataset: https://github.com/sylinrl/TruthfulQA

this is a great point.

do you know of any benchmarks doing this today?

given the acute need to evaluate models on contextual factuality, we're exploring how to create a benchmark for this purpose but prefer existing benchmarks if possible.

openai's truthfulqa[0] is close but does not focus on contextual factuality and targets a much harder problem of absolute truth.

if none exist, and people are interested in contributing, please reach out.

[0] https://github.com/sylinrl/TruthfulQA

I agree they show that you can brute-force mimick uncertainty estimates to some degree, and that the model is generally well calibrated (though on what is basically a set of trivia questions, so YMMV)... yet:

Code for https://arxiv.org/abs/2109.07958 found: https://github.com/sylinrl/TruthfulQA