promptfoo VS gpt-prompt-engineer

Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.

[0] https://github.com/typpo/promptfoo

Congrats on the launch!

I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation.

Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.

Thanks for the promptfoo mention. For anyone else who might prefer deterministic, programmatic evaluation of LLM outputs, I've been building promptfoo: https://github.com/typpo/promptfoo

Example asserts include basic string checks, regex, is-json, cosine similarity, etc.

The article from The Guardian discusses the rising issue of fake reviews generated by artificial intelligence tools, such as ChatGPT. These AI-generated reviews are becoming increasingly difficult to distinguish from genuine ones, posing new challenges for platforms like TripAdvisor, which identified 1.3 million fake reviews in 2022. AI tools are capable of producing highly plausible reviews for hotels, restaurants, and products in a variety of styles and languages. https://github.com/mshumer/gpt-prompt-engineer

It seems like a `ranking_system_prompt` prompt is used to rank the output of other prompts, which is pretty cool!

> Your job is to rank the quality of two outputs generated by different prompts. The prompts are used to generate a response for a given task. You will be provided with the task description, the test prompt, and two generations - one for each system prompt. Rank the generations in order of quality. If Generation A is better, respond with 'A'. If Generation B is better, respond with 'B'. Remember, to be considered 'better', a generation must not just be good, it must be noticeably superior to the other. Also, keep in mind that you are a very harsh critic. Only rank a generation as better if it truly impresses you more th>an the other. Respond with your ranking, and nothing else. Be fair and unbiased in your judgement.

Source: https://github.com/mshumer/gpt-prompt-engineer/blob/main/gpt...