What kind of hardware do you need to run LLaMA locally?

• llama.cpp

  775 57,463 10.0 C++

  LLM inference in C/C++

Anyhow, you'll need the latest release of llama.cpp (here is the version that supports CUDA 12.1) and you'll also need version 12.1 of CUDA toolkit (that can be found here. Note that it's over 3 GB).

• ml-ane-transformers

  22 2,461 0.0 Python

  Reference implementation of the Transformer architecture optimized for Apple Neural Engine (ANE)

  

Apple ML team released a paper and repo last year ( Ane_transformers )that shows how to optimize transformer architecture for ANE use prior to converting a PyTorch model to CoreML.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project