gateway
promptfoo
| gateway | promptfoo | |
|---|---|---|
| 6 | 5 | |
| 4,543 | 328 | |
| 7.3% | - | |
| 9.8 | 10.0 | |
| 1 day ago | 10 months ago | |
| TypeScript | TypeScript | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
gateway
- Adding a streaming run function to the Assistants API
- FLaNK Stack Weekly 22 January 2024
-
We open sourced our AI gateway written in TS
Portkey's AI Gateway is the interface between your app and hosted LLMs. It streamlines API requests to OpenAI, Anthropic, Mistral, LLama2, Anyscale, Google Gemini and more with a unified API.
- Show HN: A lightweight AI gateway to 100 models, in TS
- Open Source AI Gateway
promptfoo
-
Ollama v0.1.33 with Llama 3, Phi 3, and Qwen 110B
Jumping in because I'm a big believer in (1) local LLMs, and (2) evals specific to individual use cases.
[0] https://github.com/typpo/promptfoo
- Meta Llama 3
-
Launch HN: Talc AI (YC S23) – Test Sets for AI
Congrats on the launch!
I've been interested in automatic testset generation because I find that the chore of writing tests is one of the reasons people shy away from evals. Recently landed eval testset generation for promptfoo (https://github.com/typpo/promptfoo), but it is non-RAG so more simplistic than your implementation.
Was also eyeballing this paper https://arxiv.org/abs/2401.03038, which outlines a method for generating asserts from prompt version history that may also be useful for these eval tools.
-
GPT-Prompt-Engineer
Thanks for the promptfoo mention. For anyone else who might prefer deterministic, programmatic evaluation of LLM outputs, I've been building promptfoo: https://github.com/typpo/promptfoo
Example asserts include basic string checks, regex, is-json, cosine similarity, etc.
What are some alternatives?
rebuff - LLM Prompt Injection Detector
fill - Generative fill in 3D.
gpt-engineer - Specify what you want it to build, the AI asks for clarification, and then builds it.
llmflows - LLMFlows - Simple, Explicit and Transparent LLM Apps
ChainForge - An open-source visual programming environment for battle-testing prompts to LLMs.
VulnerableApp-facade - VulnerableApp-facade is probably most modern lightweight distributed farm of Vulnerable Applications built for handling wide range of vulnerabilities across tech stacks.
shap-e - Generate 3D objects conditioned on text or images
agenta - The all-in-one LLM developer platform: prompt management, evaluation, human feedback, and deployment all in one place.
gpt-prompt-engineer
renovate - Universal dependency automation tool.
sugarcane-ai - npm like package ecosystem for Prompts 🤖