Non-determinism in GPT-4 is caused by Sparse MoE

1. petals

   1 101 9,347 7.2 Python

   🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

   

   Could this work well with distributed solutions like petals?

   https://github.com/bigscience-workshop/petals

   I don't understand how petals can work though. I thought LLMs were typically quite monolithic.

2. SaaSHub

   www.saashub.com featured

   SaaSHub - Software Alternatives and Reviews. SaaSHub helps you find the best software and product alternatives

   

3. tensorflow

   2 233 187,290 10.0 C++

   An Open Source Machine Learning Framework for Everyone

   

   Right but that's not an inherent GPU determinism issue. It's a software issue.

   https://github.com/tensorflow/tensorflow/issues/3103#issueco... is correct that it's not necessary, it's a choice.

   Your line of reasoning appears to be "GPUs are inherently non-deterministic don't be quick to judge someone's code" which as far as I can tell is dead wrong.

   Admittedly there are some cases and instructions that may result in non-determinism but they are inherently necessary. The author should thinking carefully before introducing non-determinism. There are many scenarios where it is irrelevant, but ultimately the issue we are discussing here isn't the GPU's fault.

4. curated-transformers

   3 7 874 8.4 Python

   🤖 A PyTorch library of curated Transformer models and their composable components

   

   Yeah. In curated transformers [1] we are seeing completely deterministic output across multiple popular transformer architectures on a single GPU (there can be variance between GPUs due to different kernels).

   One non-determinism we see with a temperature of 0 is that once you have quantized weights, many predicted pieces will have the same probability, including multiple pieces with the highest probability. And then the sampler (if you are not using a greedy decoder) will sample from those pieces.

   In other words, a temperature of 0 is a poor man’s greedy decoding. (It is totally possible that OpenAI’s implementation switches to a greedy decoder with a temperature of 0).

   [1] https://github.com/explosion/curated-transformers

Suggest a related project