Mistral-8x7B-Chat

• llama.cpp

  769 56,891 10.0 C++

  LLM inference in C/C++

You'll want to try llama.cpp [1]. The set of models that it can support is expanding [2]. Folks have also written services [3] that wrap around it.

[1] https://github.com/ggerganov/llama.cpp

• llama-cpp-python

  54 6,378 9.9 Python

  Python bindings for llama.cpp

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• ollama

  192 58,943 9.9 Go

  Get up and running with Llama 3, Mistral, Gemma, and other large language models.

  

I don’t think it’s going to run in that hardware. That said you should first try the original 7b model from the official HF repo regardless.

The easiest way to get up and running is ollama.

https://huggingface.co/mistralai/Mistral-7B-v0.1

https://ollama.ai/

Suggest a related project