DeepSeek-V3 VS DeepSeek-R1

What distinguishes DeepSeek-V3 is its training efficiency—completed using only 2.664M H800 GPU hours on 14.8 trillion tokens, making it remarkably cost-effective for its size. Technical specifications are available on the GitHub page for DeepSeek-V3.

DeepSeek, known for its high-performance AI models like R1 and V3, has been a game-changer in the AI landscape. However, recent reports have highlighted issues with API instability, affecting developers and users who rely on these services. Understanding the root causes of this instability is essential for addressing and mitigating these issues.

It is not FOSS. The LLM industry has repurposed "open source" to mean "you can run the model yourself." They've released the model, but it does not meet the 'four freedoms' standard: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/LICENSE...

Typically, training parts of an AI model usually meant updating the whole thing, even if some parts didn't contribute anything, which lead to a massive waste of resources. To solve this, they introduced an Auxiliary-Loss-Free (ALS) Load Balancing. The ALS Load Balancing works by introducing a bias factor to prevent overloading one chip, while under-utilizing another (Source). This resulted in only 5% of the model's parameters being trained per-token, and around 91% cheaper cost to train than GPT 4 (GPT 4 costed $63 million to train (Source) and V3 costed $5.576 million to train. (Source))

According to the official paper, DeepSeek took only $5.6 mln to train with impressive results. This is a remarkable achievement for a large language model (LLM). In comparison, OpenAI's CEO Sam Altman admitted that training OpenAI GPT-4 took over $100 mln, not saying how much more. Some AI specialists assume that the estimation of the DeepSeek training expense is underreported. Nevertheless, the hidden gem is not how much it cost to train but how drastically it improved runtime requirements.

> If the magic values are some kind of microcode or firmware, or something else that is executed in some way, then no, it is not really open source.

To my understanding, the contents of a .safetensors file is purely numerical weights - used by the model defined in MIT-licensed code[0] and described in a technical report[1]. The weights are arguably only really "executed" to the same extent kernel weights of a gaussian blur filter would be, though there is a large difference in scale and effect.

[0]: https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inferen...

[1]: https://arxiv.org/html/2412.19437v1

But the cost is _definitely_ falling. For a recent example, see DeepSeek V3[1]. It's a model that's competitive with GPT-4, Claude Sonnet. But cost ~$6 Million to train.

This is ridiculously cheaper than what we had before. Inference is basically getting an 10x cheaper per year!

We're spending more because bigger models are worth the investment. But the "price per unit of [intelligence/quality]" is getting lower and _fast_.

Saying that models are getting more expensive is confusing the absolute value spent with the value for money.

- [1] https://github.com/deepseek-ai/DeepSeek-V3/tree/main

Both Proximal Policy Optimization (PPO) and Generalized Reinforcement Learning with Proximal Optimizer (GRPO) are the algorithm of Reinforcement Learning (RL). In this blog, I am going to explain both the algorithms in reference to a chess game. I came across these terms while reading DeepSeek R1's Research Papers.

DeepSeek, known for its high-performance AI models like R1 and V3, has been a game-changer in the AI landscape. However, recent reports have highlighted issues with API instability, affecting developers and users who rely on these services. Understanding the root causes of this instability is essential for addressing and mitigating these issues.

We kind-of have that in DeepSeek-R1-zero [1], but it has problem. From the original authors:

> With RL, DeepSeek-R1-Zero naturally emerged with numerous powerful and interesting reasoning behaviors. However, DeepSeek-R1-Zero encounters challenges such as endless repetition, poor readability, and language mixing.

A lot of these we can probably solve, but as other have pointed out we want a model that humans can converse with, not an AI for the purpose of other AI.

That said, it seems like a promising area of research:

> DeepSeek-R1-Zero demonstrates capabilities such as self-verification, reflection, and generating long CoTs, marking a significant milestone for the research community.

[1] https://github.com/deepseek-ai/DeepSeek-R1

Thank you for reading the blog. Here is the DeepSeek R1's Research Paper in case you want to check it out.

deepseek-ai/DeepSeek-R1

Benchmarks of reasoning models. Source

For R1-Zero, the RL process focused on accuracy and making sure that thinking was inside the correct tags, but it did not enforce any particular structure on the thinking itself. [1]

The pure RL approach improved reasoning, but had readability issues that led to using SFT thinking data in the final version of R1:

> DeepSeek-R1-Zero struggles with challenges like poor readability, and language mixing. To make reasoning processes more readable and share them with the open community, we explore DeepSeek-R1, a method that utilizes RL with human-friendly cold-start data.

[1] https://github.com/deepseek-ai/DeepSeek-R1/blob/main/DeepSee...

If that weren't enough, they've published the code as Open Source. Transparent. Ready to be studied, modified, and improved by anyone (Including competitors).

Access the Repository: Visit the DeepSeek R1 GitHub repository to explore the model's codebase.