github-pr-summary VS promptfoo

Excited to share my journey in creating a Telegram bot that learns Japanese and answers Q&As, all thanks to https://flows.network/ built in Rust

Maybe you can try https://flows.network/, which supports environment variables when creating an app. You can also manage the environment variables later with their UI. Click on https://flows.network/flow/createByTemplate/Telegram-ChatGPT, you can see you could set up system_prompt.

I'm wondering what will happen if we let ChatGPT review these PRs created by ChatGPT.

Yes, We made a small tool to help developer review their PR. Seems a great supplement for Sweep AI.

Build your own PR review bot in 3 minutes here: https://github.com/flows-network/github-pr-summary

Super interesting. We've been experimenting with [promptfoo](https://github.com/promptfoo/promptfoo) at my work, and this looks very similar.

In case anyone's interested in running their own benchmark across many LLMs, I've built a generic harness for this at https://github.com/promptfoo/promptfoo.

I encourage people considering LLM applications to test the models on their _own data and examples_ rather than extrapolating general benchmarks.

This library supports OpenAI, Anthropic, Google, Llama and Codellama, any model on Replicate, and any model on Ollama, etc. out of the box. As an example, I wrote up an example benchmark comparing GPT model censorship with Llama models here: https://promptfoo.dev/docs/guides/llama2-uncensored-benchmar.... Hope this helps someone.