Chat with, and help host, a free community LLM "horde"

• koboldcpp

  180 3,749 10.0 C++

  A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

It works like this:

- The AI Horde hosts a web app (Kobold Lite) geared towards LLM chat and RP. Its mature, predating LLAMA and GPT 3.5 and largely developed when the RP community was running GPT-J finetunes. There are mature desktop apps that can access this API as well.

- The user sets the chat syntax/format and picks a LLM host (or multiple hosts).

- These hosts run simple API endpoints from any PC for Horde users to access. The backends de-joure are koboldcpp, a frontend for llama.cpp which is excellent, portable and literally one click, and KoboldAI, with the very fast and vram-efficient exllamav2 backend:

https://github.com/LostRuins/koboldcpp

• KoboldAI

  41 327 9.5 C++

https://github.com/henk717/KoboldAI

- Hosts pick a quantized community LLM to run, which is (IMO) the real magic of this system. Cloud services tend to run generic Llama chat/instruct models, OpenAI API models, or maybe a single proprietary finetune, but the Llama/Mistral finetuning community is red hot. New finetines and crazy merges/hybrids that outperform llama-chat in specific tasks (mostly Chat/Story/RP) come out every day, and each one has a different "flavor" and format:

https://huggingface.co/models?sort=modified&search=mistral+g...

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project