big-AGI VS S-LoRA

You could run a local copy of big-agi and edit this file to use the dated model.

As ChatGPT counts tokens for both inputs and output, you'd have to be thoughtful for the input side as well so this is where the tool would come in handy within the same chat session. I don't think it remembers separate chat sessions so yeah without access to history, you'd need to either start a new chat session each time or setup your own ChatGPT instance and use your own OpenAI API token and be billed. For my own ChatGPT instance I use awesome app at https://github.com/enricoros/nextjs-chatgpt-app which has a demo on the readme page too.

nice, maybe you can make the initializing prompts in

https://github.com/enricoros/nextjs-chatgpt-app/blob/main/pa...

transparent to the user and even changeable by them.

You can also batch requests using different LoRAs. See "S-LoRA: Serving Thousands of Concurrent LoRA Adapters". https://arxiv.org/abs/2311.03285

Depending on what you mean by "production" you'll probably want to look at "real" serving implementations like HF TGI, vLLM, lmdeploy, Triton Inference Server (tensorrt-llm), etc. There are also more bespoke implementations for things like serving large numbers of LoRA adapters[0].

These are heavily optimized for more efficient memory usage, performance, and responsiveness when serving large numbers of concurrent requests/users in addition to things like model versioning/hot load/reload/etc, Prometheus metrics, things like that.

One major difference is at this level a lot of the more aggressive memory optimization techniques and support for CPU aren't even considered. Generally speaking you get GPTQ and possibly AWQ quantization + their optimizations + CUDA only. Their target users and their use cases are often using A100/H100 and just trying to need fewer of them. Support for lower VRAM cards, older CUDA compute architectures, etc come secondary to that (for the most part).

[0] - https://github.com/S-LoRA/S-LoRA