gpt-engineer VS promptfoo

2️⃣ GPT-engineer 🧪

you should check out this:

https://github.com/AntonOsika/gpt-engineer

3. https://github.com/AntonOsika/gpt-engineer

Framework to Generate HQ code (I'm a mod at gpt-engineer)

BTW, GPT-Engineer is openly collecting all of your data: user prompts and other metadata. And they were even defending it until they received some strong responses from the community: https://github.com/AntonOsika/gpt-engineer/issues/415 They now explicitly ask for consent regarding user data, but can we really trust their motives?

You might be interested in gpt-engineer https://github.com/AntonOsika/gpt-engineer

Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.

[0] https://github.com/typpo/promptfoo

Congrats on the launch!

I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation.

Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.

Thanks for the promptfoo mention. For anyone else who might prefer deterministic, programmatic evaluation of LLM outputs, I've been building promptfoo: https://github.com/typpo/promptfoo

Example asserts include basic string checks, regex, is-json, cosine similarity, etc.