GPTCache VS llama.cpp

Just found this: https://github.com/zilliztech/GPTCache which seems to address this idea/issue.

21. GPTCache | Github | tutorial

Check this out. Built on a vector database (https://github.com/milvus-io/milvus) and a semantic cache (https://github.com/zilliztech/GPTCache)

https://osschat.io/

With the proliferation of large models, an increasing number of enterprises and individual developers are now developing applications based on these models. As such, it is worth considering whether large model caching is necessary during the development process.

Our project: https://github.com/zilliztech/GPTCache

Maybe [GPTCache](https://github.com/zilliztech/GPTCache) can make it more attractive, because similar problems can be less expensive, and can also be responded to faster. Of course, the specific configuration needs to be based on real usage scenarios.

if you can compile stuff, then looking at llama.cpp (what ollama uses) is also interesting: https://github.com/ggerganov/llama.cpp

the server is here: https://github.com/ggerganov/llama.cpp/tree/master/examples/...

And you can search for any GGUF on huggingface

Yes, Metal seems to allow a maximum of 1/2 of the RAM for one process, and 3/4 of the RAM allocated to the GPU overall. There’s a kernel hack to fix it, but that comes with the usual system integrity caveats. https://github.com/ggerganov/llama.cpp/discussions/2182

For anyone interested in exploring this, llama.cpp has an example implementation here:

https://github.com/ggerganov/llama.cpp/tree/master/examples/...

Getting started with LLMs can be intimidating. In this tutorial we will show you how to fine-tune a large language model using LoRA, facilitated by tools like llama.cpp and KitOps.

well https://github.com/ggerganov/llama.cpp/issues/6849