promptfoo
prompt-engineering
| promptfoo | prompt-engineering | |
|---|---|---|
| 20 | 18 | |
| 2,921 | 8,028 | |
| 23.7% | 2.8% | |
| 9.9 | 5.1 | |
| 6 days ago | 7 months ago | |
| TypeScript | ||
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
promptfoo
- Google CodeGemma: Open Code Models Based on Gemma [pdf]
- AI Infrastructure Landscape
- Promptfoo – Testing and Evaluation for LLMs
-
Show HN: Prompt-Engineering Tool: AI-to-AI Testing for LLM
Super interesting. We've been experimenting with [promptfoo](https://github.com/promptfoo/promptfoo) at my work, and this looks very similar.
- GitHub – promptfoo/promptfoo: Test your prompts
-
I asked 60 LLMs a set of 20 questions
In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo.
I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks.
This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Ollama, etc. out of the box. As an example, I wrote up an example benchmark comparing GPT model censorship with Llama models here: https://promptfoo.dev/docs/guides/llama2-uncensored-benchmar.... Hope this helps someone.
- Ask HN: Prompt Manager for Developers
- DeepEval – Unit Testing for LLMs
- Show HN: Knit – A Better LLM Playground
- Show HN: CLI for testing and evaluating LLM outputs
prompt-engineering
- Ask HN: Any good collection of writing prompts for GPT 3.5/4?
-
Show HN: LLM Agent Paper List
An agent is a style of prompt that lets LLMs act as reasoning engines. It's also known as the ReAct pattern (which engineers are avoiding using for namespace collision reasions).
You can read a good intro example here: https://github.com/brexhq/prompt-engineering#react
- FLaNK Stack Weekly for 20 June 2023
-
What are your long-term career goals?
Well, if developers get replaced by AI, then who are the managers going to manage :). I personally don't think AI is just going to replace us. The way we work will continue to change as new AI tools come out. I'm taking time to tinker with new tools and seeing how others do as well (e.g., I found Brex's tips and tricks for working with LLMs very insightful: https://github.com/brexhq/prompt-engineering).
-
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT
I recognize there's plenty of catnip here when it comes to calling this "engineering" or not, however, whatever you want to call it (prompt fiddling?), the techniques are crucial if you want to achieve reasonably consistent output from current-state LLMs. As models improve concerns about context window limitations will be reduced and it will be easier to discern user intent.
These are good straight-to-the-point guides:
- Prompt Engineering by BrexHQ: https://github.com/brexhq/prompt-engineering
- OpenAI guidance: https://help.openai.com/en/articles/6654000-best-practices-f...
- https://devblogs.microsoft.com/dotnet/gpt-prompt-engineering...
- (great examples): https://www.deeplearning.ai/short-courses/chatgpt-prompt-eng...
tl;dr:
-
(2/2) May 2023
Brex's Prompt Engineering Guide (https://github.com/brexhq/prompt-engineering)
- GitHub - brexhq/prompt-engineering: Tips and tricks for working with Large Language Models like OpenAI's GPT-4.
- Brex’s Prompt Engineering Guide
What are some alternatives?
shap-e - Generate 3D objects conditioned on text or images
Prompt-Engineering-Guide - 🐙 Guides, papers, lecture, notebooks and resources for prompt engineering
WizardLM - Family of instruction-following LLMs powered by Evol-Instruct: WizardLM, WizardCoder and WizardMath
FinGPT - FinGPT: Open-Source Financial Large Language Models! Revolutionize 🔥 We release the trained model on HuggingFace.
chat-ui - Open source codebase powering the HuggingChat app
tree-of-thoughts - Plug in and Play Implementation of Tree of Thoughts: Deliberate Problem Solving with Large Language Models that Elevates Model Reasoning by atleast 70%
litellm - Call all LLM APIs using the OpenAI format. Use Bedrock, Azure, OpenAI, Cohere, Anthropic, Ollama, Sagemaker, HuggingFace, Replicate (100+ LLMs)
chathub - All-in-one chatbot client
ChainForge - An open-source visual programming environment for battle-testing prompts to LLMs.
canal - 阿里巴巴 MySQL binlog 增量订阅&消费组件
WizardVicunaLM - LLM that combines the principles of wizardLM and vicunaLM
modelscope - ModelScope: bring the notion of Model-as-a-Service to life.