Apple Silicon Llama 7B running in docker?

• nitro

  7 1,572 9.8 C++

  An inference server on top of llama.cpp. OpenAI-compatible API, queue, & scaling. Embed a prod-ready, local inference engine in your apps. Powers Jan (by janhq)

  

• bionic-gpt

  11 1,590 9.8 Rust

  BionicGPT is an on-premise replacement for ChatGPT, offering the advantages of Generative AI while maintaining strict data confidentiality

  

I'm the maintainer of https://github.com/bionic-gpt/bionic-gpt and we have a nice install option for windows and Linux but nothing for Apple Silicon.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llama.cpp

  769 56,891 10.0 C++

  LLM inference in C/C++

• ollama

  195 58,943 9.9 Go

  Get up and running with Llama 3, Mistral, Gemma, and other large language models.

  

How important is using docker? If the main concern is ease of installation then look into Ollama, as the installation process is the same as any other Mac app (download and drag .app to Applications). It has options to interact through terminal or a browser.

Suggest a related project