How do I utilize these quantized models being uploaded?

• llama.cpp

  777 57,984 10.0 C++

  LLM inference in C/C++

The actual usage is different. For on-CPU inference with ggmlv3.bin models you use llama.cpp or one of its derivatives. GPU-oriented quantization such as GPTQ are also available, but, as I don't have a compatible GPU, I ignore them.

• koboldcpp

  180 3,887 10.0 C++

  A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• text-generation-webui

  876 36,827 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

• ctransformers

  4 1,707 8.6 C

  Python bindings for the Transformer models implemented in C/C++ using GGML library.

You can also use ctransformers with the ggml models if you want to use python rather than c++.

Suggest a related project