gpt-discord-bot VS llama-cpp-python

I’ve set up an initial bot using the gpt-discord-bot package. I’m wondering how to train it with more data like an entire PDF for example instead of editing the instructions in config.yaml. Also, how would I go about hosting this so it can run constantly, do I just run in on a standard Linux server with certain firewall settings?

Hey guys, I'm new to AI chat bots but eager to learn. I'm playing around with [gpt-discord-bot](https://github.com/openai/gpt-discord-bot) and have a few questions to get me up and running if someone has time:

And here's a Discord bot that currently works with it that you may be able to learn from: https://github.com/openai/gpt-discord-bot

Here's the page that tells you how to do it but you'll need some programming knowledge in python to get it to work. it's not just something you can invite to your server. https://github.com/openai/gpt-discord-bot

This one, right here >> https://github.com/openai/gpt-discord-bot Follow the instruction and you'll get it working.

Go to the official OpenAI Discord - https://discord.gg/openai and then go to #gpt-discord-bot and that'll send you to https://github.com/openai/gpt-discord-bot to get the code. I'm running the code on a RaspberryPi but originally I ran it on my MacBook. Super easy to setup. Just needs an API key from OpenAI you can get here: https://beta.openai.com/account/api-keys once you give them a credit card for billing https://beta.openai.com/account/billing/overview and you can set limits on what they charge you. It's honestly super cheap. For Discord you just need a server you own to invite the bot to and of course Discord lets you setup a server for free.

I'll share the core bit that took a while to figure out the right format, my main script is a hot mess using embeddings with SentenceTransformer, so I won't share that yet. E.g: last night I did a PR for llama-cpp-python that shows how Phi might be used with JSON only for the author to write almost exactly the same code at pretty much the same time. https://github.com/abetlen/llama-cpp-python/pull/1184

Python Bindings for llama.cpp

If the model could be made to work with llama.cpp, then https://github.com/abetlen/llama-cpp-python might be more compact. llama.cpp only supports a limited list of model types though.

https://github.com/abetlen/llama-cpp-python has a web server mode that replicates openai's API iirc and the readme shows it has docker builds already.

LocalAI https://localai.io/ and LMStudio https://lmstudio.ai/ both have fairly complete OpenAI compatibility layers. llama-cpp-python has a FastAPI server as well: https://github.com/abetlen/llama-cpp-python/blob/main/llama_... (as of this moment it hasn't merged GGUF update yet though)

I went with Python, llama-cpp-python, since my goal is just to get a small project up and running locally.

I see you’re using gpt4all; do you have a supported way to change the model being used for local inference?

A number of apps that are designed for OpenAI’s completion/chat APIs can simply point to the endpoints served by llama-cpp-python [0], and function in (largely) the same way, while supporting the various models and quants supported by llama.cpp. That would allow folks to run larger models on the hardware of their choice (including Apple Silicon with Metal acceleration) or using other proxies like openrouter.io.

[0]: https://github.com/abetlen/llama-cpp-python