How to - Local LLMs ?

• text-generation-webui

  876 37,224 9.9 Python

  A Gradio web UI for Large Language Models. Supports transformers, GPTQ, AWQ, EXL2, llama.cpp (GGUF), Llama models.

weird, working for me. if you want text, just read the info: https://github.com/oobabooga/text-generation-webui

• mlc-llm

  89 17,302 9.9 Python

  Enable everyone to develop, optimize and deploy AI models natively on everyone's devices.

  

check this out: mlc-llm

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• pyllamacpp

  4 59 7.0 C++

  Python bindings for llama.cpp

For me, the easiest way was to start was to use https://github.com/abdeladim-s/pyllamacpp. If you have Python installed, do `pip install numpy pyllamacpp`, download compatible model file (links are in README.md) and simply run with `pyllamacpp model.file`.

Suggest a related project