promptfoo
gateway
promptfoo | gateway | |
---|---|---|
5 | 7 | |
328 | 4,830 | |
- | 5.9% | |
10.0 | 9.8 | |
11 months ago | 6 days ago | |
TypeScript | TypeScript | |
MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
promptfoo
-
Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.
[0] https://github.com/typpo/promptfoo
- Meta Llama 3
-
Launch HN: Talc AI (YC S23) – Test Sets for AI
Congrats on the launch!
I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation.
Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.
-
GPT-Prompt-Engineer
Thanks for the promptfoo mention. For anyone else who might prefer deterministic, programmatic evaluation of LLM outputs, I've been building promptfoo: https://github.com/typpo/promptfoo
Example asserts include basic string checks, regex, is-json, cosine similarity, etc.
gateway
- Adding a streaming run function to the Assistants API
- FLaNK Stack Weekly 22 January 2024
-
We open sourced our AI gateway written in TS
Portkey's AI Gateway is the interface between your app and hosted LLMs. It streamlines API requests to OpenAI, Anthropic, Mistral, LLama2, Anyscale, Google Gemini and more with a unified API.
- Show HN: A lightweight AI gateway to 100 models, in TS
- Open Source AI Gateway
What are some alternatives?
rebuff - LLM Prompt Injection Detector
gpt-engineer - Specify what you want it to build, the AI asks for clarification, and then builds it.
fill - Generative fill in 3D.
ChainForge - An open-source visual programming environment for battle-testing prompts to LLMs.
llmflows - LLMFlows - Simple, Explicit and Transparent LLM Apps
shap-e - Generate 3D objects conditioned on text or images
VulnerableApp-facade - VulnerableApp-facade is probably most modern lightweight distributed farm of Vulnerable Applications built for handling wide range of vulnerabilities across tech stacks.
sugarcane-ai - npm like package ecosystem for Prompts 🤖
agenta - The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.
plandex - AI driven development in your terminal. Designed for large, real-world tasks.
renovate - Home of the Renovate CLI: Cross-platform Dependency Automation by Mend.io