TinyLlama project aims to pretrain a 1.1B Llama model on 3T tokens

• TinyLlama

  14 6,818 8.7 Python

  The TinyLlama project is an open endeavor to pretrain a 1.1B Llama model on 3 trillion tokens.

• llama.cpp

  773 56,891 10.0 C++

  LLM inference in C/C++

Could this be used as a source of speculative tokens for larger llama models?, as per https://github.com/ggerganov/llama.cpp/pull/2926

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• airoboros

  8 948 8.7 Python

  Customizable implementation of the self-instruct paper.

• hydra-moe

  2 395 8.4 Python
  

Thanks. Yes, I've seen airoboros, it aims to use a mixture of fine-tunes of the base model if I recall correctly. Not a truly pre-trained MOE, but could be useful.

Hydra, is this it? https://github.com/SkunkworksAI/hydra-moe

Suggest a related project