My largest ever quants, GPT 3 sized! BLOOMZ 176B and BLOOMChat 1.0 176B

• llama.cpp

  773 56,891 10.0 C++

  LLM inference in C/C++

So I am looking at running this with llama.cpp with metal shaders (Mac M1 Ultra, 128 Gb) but running into a conversion problem. I can get tokenizer.json to tokenizer.model using this method, but can't convert the model to the q4_0 .bin that llama.cpp uses.

• bloomz.cpp

  4 807 6.0 C

  C++ implementation for BLOOM

Possibly. There's a llama.cpp fork called bloomz.cpp but it's not been updated in 2 months. So it's not going to support any of the fancy new quantisation methods, performance improvements, GPU acceleration, etc.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• lm-evaluation-harness

  34 5,070 9.9 Python

  A framework for few-shot evaluation of language models.

  

Hey u/The-Bloke Appreciate the quants! What is the degradation on the some benchmarks. Have you seen https://github.com/EleutherAI/lm-evaluation-harness. 3-bit and 2-bit quant will really be pushing it. I don't see a ton of evaluation results on the quants and nice to see a before and after.

• ggml

  3 1 10.0 C

  Discontinued Tensor library for machine learning (by the-crypt-keeper)

You need my ggml fork until #343 is merged into ggml to use it.

• ggml

  69 9,725 9.8 C

  Tensor library for machine learning

You need my ggml fork until #343 is merged into ggml to use it.

• SaaSHub

  www.saashub.com featured

  SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

  

Suggest a related project