empirical-philosophy VS guardrails

One way that I've been framing this in my head (and in an application I'm building) is that gpt-3 will be useful for analytic tasks where as gpt-4 will be required for synthetic tasks. I'm using "analytic" and "synthetic" in the same way as in this writeup https://github.com/williamcotton/empirical-philosophy/blob/m...

Hey Simon! I've been digging your writings on LLMs lately.

I've been having some decent luck with some of the approaches that I've discussed in the following articles and projects:

From Prompt Alchemy to Prompt Engineering: An Introduction to Analytic Augmentation: https://github.com/williamcotton/empirical-philosophy/blob/m...

https://www.williamcotton.com/articles/writing-web-applicati...

https://github.com/williamcotton/transynthetical-engine

I'd love to hear your thoughts on the matter!

You’re not actually doing any research.

Here is my research: https://github.com/williamcotton/empirical-philosophy/blob/m...

It is clear that analytic augmentations will result in more factual information.

Your claims are unfounded and untested.

Take a look at

https://github.com/williamcotton/empirical-philosophy/blob/m...

https://langchain.readthedocs.io/en/latest/

They can be taught!

I've been developing a methodology around prompt engineering that I have found very useful:

https://github.com/williamcotton/empirical-philosophy/blob/m...

A few more edits and it's ready for me to submit to HN and then get literally no further attention!

That's not a rebuttable of a claim that Bing is more accurate.

A proper rebuttable would involve empirical evidence that Bing is no more accurate than other LLM tools that do not add analytical augmentations such as search results to their prompts.

Based on empirical evidence, I find that analytical augmentations do indeed result in more accurate results:

https://github.com/williamcotton/empirical-philosophy/blob/m...

You're absolutely correct, and I agree that there's potentially a risk of quality loss. But likewise, since these are all intrinsically linked, it may be possible to leverage strength by combining these tasks. I'm unaware of a paper reviewing the reliability and/or performance of LLMs in this specific scenario. If you find any, do share :) With regards to generating JSON responses - there are simple ways to nudge the model and even validate it, using libraries such as https://github.com/promptslab/Promptify, https://github.com/eyurtsev/kor and https://github.com/ShreyaR/guardrails

There are a couple different approaches:

- Use multi-shot prompting with something like guardrails to try prompting a commercial model until it works. [1]

- Use a local model with something with a final layer that steers token selection towards syntactically valid tokens [2]

[1] https://github.com/ShreyaR/guardrails

[2] "Structural Alignment: Modifying Transformers (like GPT) to Follow a JSON Schema" @ https://github.com/newhouseb/clownfish.

👍 validate and correct the outputs of LLMs using guardrails

not vouching for it, but I know this is at least a thing that exists and I like the general idea: https://github.com/shreyar/guardrails