rubygpt VS GPTCache

I've got an IRC channel full of ChatGPT bots that answer questions from different perspectives (using [1] if anyone wants something similar). This comment made me realize it'd probably be extremely interesting to also let them debate between themselves to come to a singular answer -- and it'd also be interesting to see which of the popular language models most often "yield" to each other in bot-vs-bot battles (rap or otherwise). Probably a lot of exciting meta-analytics on models to be done here.

[1] https://github.com/drusepth/rubygpt/blob/master/council.rb

GPTCache — an open source library built specifically for caching LLM responses. Pretty flexible and worth looking at if you're rolling your own pipeline.

Production Sampling: In highload systems, evaluating 100% of traffic via GPT-4o is financial suicide. Use sampling (1–5%). Additionally: implement judge caching (GPT cache, LangChain cache, or vLLM prefix caching). This is especially effective when users ask similar questions — the same prompt+context can be evaluated multiple times, but you pay only once.

Just found this: https://github.com/zilliztech/GPTCache which seems to address this idea/issue.

21. GPTCache | Github | tutorial

Check this out. Built on a vector database (https://github.com/milvus-io/milvus) and a semantic cache (https://github.com/zilliztech/GPTCache)

https://osschat.io/

With the proliferation of large models, an increasing number of enterprises and individual developers are now developing applications based on these models. As such, it is worth considering whether large model caching is necessary during the development process.

Our project: https://github.com/zilliztech/GPTCache

Maybe [GPTCache](https://github.com/zilliztech/GPTCache) can make it more attractive, because similar problems can be less expensive, and can also be responded to faster. Of course, the specific configuration needs to be based on real usage scenarios.