Mistral 7B,The complete Guide of the Best 7B model

• petals

  99 8,763 8.3 Python

  🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

  

https://github.com/bigscience-workshop/petals

Inference only: https://lite.koboldai.net/

• aphrodite-engine

  2 634 9.7 Python

  PygmalionAI's large-scale inference engine

  

If you want to host for the AI Horde, Aphrodite has the most GPU throughput, and Koboldcpp is the easiest and quite flexible:

https://github.com/PygmalionAI/aphrodite-engine

• Scout Monitoring

  www.scoutapm.com featured

  Free Django app performance insights with Scout Monitoring. Get Scout setup in minutes, and let us sweat the small stuff. A couple lines in settings.py is all you need to start monitoring your apps. Sign up for our free tier today.

  

• koboldcpp

  180 4,078 10.0 C++

  A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

https://github.com/LostRuins/koboldcpp

I havent messed with Petals yet either, but swarm finetuning is very appealing... If thats even implemented atm.

Suggest a related project