promptfoo

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality. [Moved to: https://github.com/promptfoo/promptfoo] (by typpo)

Promptfoo Alternatives

Similar projects and alternatives to promptfoo

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a better promptfoo alternative or higher similarity.

promptfoo reviews and mentions

Posts with mentions or reviews of promptfoo. We have used some of these posts to build our list of alternatives and similar projects. The last one was on 2024-04-28.
  • Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
    11 projects | news.ycombinator.com | 28 Apr 2024
    Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.

    [0] https://github.com/typpo/promptfoo

  • Meta Llama 3
    10 projects | news.ycombinator.com | 18 Apr 2024
  • Launch HN: Talc AI (YC S23) – Test Sets for AI
    1 project | news.ycombinator.com | 18 Jan 2024
    Congrats on the launch!

    I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation.

    Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.

  • GPT-Prompt-Engineer
    6 projects | news.ycombinator.com | 10 Jul 2023
    Thanks for the promptfoo mention. For anyone else who might prefer deterministic, programmatic evaluation of LLM outputs, I've been building promptfoo: https://github.com/typpo/promptfoo

    Example asserts include basic string checks, regex, is-json, cosine similarity, etc.

  • A note from our sponsor - SaaSHub
    www.saashub.com | 1 May 2024
    SaaSHub helps you find the best software and product alternatives Learn more →

Stats

Basic promptfoo repo stats
5
328
10.0
10 months ago

typpo/promptfoo is an open source project licensed under MIT License which is an OSI approved license.

The primary programming language of promptfoo is TypeScript.


Sponsored
SaaSHub - Software Alternatives and Reviews
SaaSHub helps you find the best software and product alternatives
www.saashub.com