Mistral AI Launches New 8x22B Moe Model

• vllm

  31 18,571 9.9 Python

  A high-throughput and memory-efficient inference and serving engine for LLMs

  

The easiest is to use vllm (https://github.com/vllm-project/vllm) to run it on a Couple of A100's, and you can benchmark this using this library (https://github.com/EleutherAI/lm-evaluation-harness)

• lm-evaluation-harness

  34 5,070 9.9 Python

  A framework for few-shot evaluation of language models.

  

The easiest is to use vllm (https://github.com/vllm-project/vllm) to run it on a Couple of A100's, and you can benchmark this using this library (https://github.com/EleutherAI/lm-evaluation-harness)

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• llamafile

  34 14,839 9.6 C++

  Distribute and run LLMs with a single file.

  

I think the llamafile[0] system works the best. Binary works on the command line or launches a mini webserver. Llamafile offers builds of Mixtral-8x7B-Instruct, so presumably they may package this one up as well (potentially a quantized format).

You would have to confirm with someone deeper in the ecosystem, but I think you should be able to run this new model as is against a llamafile?

[0] https://github.com/Mozilla-Ocho/llamafile

• plandex-ai

  1 - -

I built one using GPT-4[1]. It's not perfect but is working quite well and is now being used by hundreds of users, apart from me, to work on real, non-toy tasks. For example, I used it to build most of a production-ready AWS infrastructure (and accompanying deploy script) with the AWS CDK.

I want to add Mistral support soon, probably via together.ai or a similar service.

1 - https://github.com/plandex/plandex-ai

Suggest a related project