guardrails VS ghostwheel

You're absolutely correct, and I agree that there's potentially a risk of quality loss. But likewise, since these are all intrinsically linked, it may be possible to leverage strength by combining these tasks. I'm unaware of a paper reviewing the reliability and/or performance of LLMs in this specific scenario. If you find any, do share :) With regards to generating JSON responses - there are simple ways to nudge the model and even validate it, using libraries such as https://github.com/promptslab/Promptify, https://github.com/eyurtsev/kor and https://github.com/ShreyaR/guardrails

There are a couple different approaches:

- Use multi-shot prompting with something like guardrails to try prompting a commercial model until it works. [1]

- Use a local model with something with a final layer that steers token selection towards syntactically valid tokens [2]

[1] https://github.com/ShreyaR/guardrails

[2] "Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema" @ https://github.com/newhouseb/clownfish.

👍 validate and correct the outputs of LLMs using guardrails

not vouching for it, but I know this is at least a thing that exists and I like the general idea: https://github.com/shreyar/guardrails

big fan of fulcro logic, but isn’t this almost identical to https://github.com/gnl/ghostwheel ?

if all you care about is information / documentation, and maybe validation, then spec or malli are fine. using spec during runtime to simulate a type system is a poor choice in most cases, because you are doing runtime type checking (something akin to dependent types or even esoteric contracts that are hard/impossible to express statically). There are some libs that help bolt a typed facade (using spec) onto the familiar defn / fn, https://github.com/gnl/ghostwheel https://github.com/Provisdom/defn-spec .