gpt-prompt-engineer VS ChainForge

The article from The Guardian discusses the rising issue of fake reviews generated by artificial intelligence tools, such as ChatGPT. These AI-generated reviews are becoming increasingly difficult to distinguish from genuine ones, posing new challenges for platforms like TripAdvisor, which identified 1.3 million fake reviews in 2022. AI tools are capable of producing highly plausible reviews for hotels, restaurants, and products in a variety of styles and languages. https://github.com/mshumer/gpt-prompt-engineer

It seems like a `ranking_system_prompt` prompt is used to rank the output of other prompts, which is pretty cool!

> Your job is to rank the quality of two outputs generated by different prompts. The prompts are used to generate a response for a given task. You will be provided with the task description, the test prompt, and two generations - one for each system prompt. Rank the generations in order of quality. If Generation A is better, respond with 'A'. If Generation B is better, respond with 'B'. Remember, to be considered 'better', a generation must not just be good, it must be noticeably superior to the other. Also, keep in mind that you are a very harsh critic. Only rank a generation as better if it truly impresses you more th>an the other. Respond with your ranking, and nothing else. Be fair and unbiased in your judgement.

Source: https://github.com/mshumer/gpt-prompt-engineer/blob/main/gpt...

This seems like a sheets implementation of something like ChainForge (https://github.com/ianarawjo/ChainForge). Curious that Anthropic is entering the LLMOps tooling space ---this definitely comes as a surprise to me, as both OpenAI and HuggingFace seem to avoid building prompt engineering tooling themselves. Is this a business strategy of Anthropic's? An experiment? Regardless, it's very cool to see a company like them throw their hat into the LLMOps space beyond being a model provider. Interested to see what comes next.

ChainForge has similar functionality for comparing : https://github.com/ianarawjo/ChainForge

LocalAI creates a GPT-compatible HTTP API for local LLMs: https://github.com/go-skynet/LocalAI

Is it necessary to have an HTTP API for each model in a comparative study?

I think you should probably mention that its source is available! [0]

I don't personally have a need for this right now, but I can really see the use for the parameterised queries, as well as comparisons across models.

Thanks for your efforts!

0: https://github.com/ianarawjo/ChainForge

No problem! I guess I will make a plug myself --we've been working on a similar 'prompt engineering', ChainForge (https://github.com/ianarawjo/ChainForge). It's targeted towards slightly different users and use cases than promptfoo --probably more geared towards early-stage, 'quick-and-dirty' prompting explorations of differences between prompts and models for less experience programmers, versus the kind of continuious benchmarking and verification testing that promptfoo offers.

I particularly like promptfoo's support for CI, which I haven't seen anywhere else, and is very important for developers pushing prompts into production (esp since OpenAI keeps updating their models every few months...).