Best way to use AMD CPU and GPU

• serge

  40 5,543 9.8 Svelte

  A web interface for chatting with Alpaca through llama.cpp. Fully dockerized, with an easy to use API.

  

Serge made it really easy for me to get started, but it all CPU-based.

• mlc-llm

  89 16,955 9.9 Python

  Enable everyone to develop, optimize and deploy AI models natively on everyone's devices.

  

MLC LLM looks like an easy option to use my AMD GPU.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llama.cpp

  773 56,891 10.0 C++

  LLM inference in C/C++

Llama.cpp seems like it can use both CPU and GPU, but I haven't quite figured that out yet.

• koboldcpp

  180 3,817 10.0 C++

  A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

You can try Koboldcpp with CLblast from this repo: https://github.com/LostRuins/koboldcpp/releases It allows to offload several layers to GPU with significant boost of prompt processing speed and inference speed.

• exllama

  64 2,594 9.0 Python

  A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights.

As mentioned, exllama is the way to go. Once you install ROCm and the official ROCm PyTorch you are ready to go. A 16GB 6800XT will support running 13B 4-bit GPTQs with full context. Spare_Side just posted a report w/ the same GPU.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project