Reaching LLaMA2 Performance with 0.1M Dollars

• llama

  184 53,053 8.1 Python

  Inference code for Llama models

  

> JetMoE-8B is trained with less than $ 0.1 million1 cost but outperforms LLaMA2-7B from Meta AI, who has multi-billion-dollar training resources. LLM training can be much cheaper than people generally thought.

They want you to read this as "we spent $100k compared to Meta's spending billions", but that's not actually what this says. It says that they spent $100k and Meta has the resources to spend billions if they wanted to.

We don't know what Facebook spent on training LLaMA 2, but they say that it took them 184320 A100-80GB GPU-hours to train the 7B model [0]. AWS charges $14.46/hour for an instance that has 8 of those [1], which amounts to $1.81/GPU/hr.

At that rate and assuming they paid something resembling AWS's list price, LLaMA 2 7B cost ~$333k. That's more than $100k, but not by orders of magnitude, and it's likely that Facebook wasn't paying the full price AWS is charging today.

[0] https://github.com/meta-llama/llama/blob/main/MODEL_CARD.md#...

[1] https://aws.amazon.com/ec2/instance-types/p4/

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project