prompt-engineering VS promptfoo

An agent is a style of prompt that lets LLMs act as reasoning engines. It's also known as the ReAct pattern (which engineers are avoiding using for namespace collision reasions).

You can read a good intro example here: https://github.com/brexhq/prompt-engineering#react

Well, if developers get replaced by AI, then who are the managers going to manage :). I personally don't think AI is just going to replace us. The way we work will continue to change as new AI tools come out. I'm taking time to tinker with new tools and seeing how others do as well (e.g., I found Brex's tips and tricks for working with LLMs very insightful: https://github.com/brexhq/prompt-engineering).

I recognize there's plenty of catnip here when it comes to calling this "engineering" or not, however, whatever you want to call it (prompt fiddling?), the techniques are crucial if you want to achieve reasonably consistent output from current-state LLMs. As models improve concerns about context window limitations will be reduced and it will be easier to discern user intent.

These are good straight-to-the-point guides:

- Prompt Engineering by BrexHQ: https://github.com/brexhq/prompt-engineering

- OpenAI guidance: https://help.openai.com/en/articles/6654000-best-practices-f...

- https://devblogs.microsoft.com/dotnet/gpt-prompt-engineering...

- (great examples): https://www.deeplearning.ai/short-courses/chatgpt-prompt-eng...

tl;dr:

Brex's Prompt Engineering Guide (https://github.com/brexhq/prompt-engineering)

Super interesting. We've been experimenting with [promptfoo](https://github.com/promptfoo/promptfoo) at my work, and this looks very similar.

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo.

I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks.

This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Ollama, etc. out of the box. As an example, I wrote up an example benchmark comparing GPT model censorship with Llama models here: https://promptfoo.dev/docs/guides/llama2-uncensored-benchmar.... Hope this helps someone.