DeepLearningExamples VS Megatron-LM

GitHub Used: https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2

Disclosure: I used to work on Google Cloud.

I dunno, their A100 results took about 20-30 minutes on 8 x A100s [1]. 8xA100s is like $24/hr on GCP at on-demand rates.

The efficiency was okay but not linear, so if you were more cost constrained you might go with 1xA100 for $3/hr and have ~2.5hr training times.

Getting that performance out of a GPU is more challenging than getting access to the GPUs. All the major cloud providers offer them.

(Nit: GCP deployed the 40 GiB cards rather than the later 80 GiB parts, but let's ignore that).

but it often doesn't matter

[1] https://github.com/NVIDIA/DeepLearningExamples/tree/master/P...

Entrypoint.py file in tacotron2 folder: source code

For those asking about differences to xVASynth, the models trained with xVASynth are the FastPitch models (https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch). As a quick explainer:

There's another separate tool set from Nvidia that's on GitHub that the creator used to train the models. I'm not going to pretend like I understand it, but you can find it here.

The Nvidia's implementation of BERT has a long way to go (I don't know about the implementations of input independent gradient computations in their backprop). But, there are scaled benchmarks on DGX A100's -https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/LanguageModeling/BERT

https://github.com/NVIDIA/Megatron-LM

This is probably a good baseline to start thinking about LLM training at scale.

This 20B model was trained on the same datasets as its predecessor, aptly named The Pile. Furthermore, the libraries Megatron and DeepSpeed were used to achieve better computing resource utilization, and eventually GPT-NeoX evolved into its own framework for training other LLMs. It was used, for example, as the foundation for Llemma, an open-source model specializing on theorem proving.

Deepspeed Megatron-LM

GPU cluster scaling has come a long way. Just checkout the scaling plot here: https://github.com/NVIDIA/Megatron-LM

I found their code that the softmax function conduct all-reduce before they work.

Google's PaLM, Facebook's LLaMA, Nvidia's Megatron, I am missing some surely and Apple sure has something cooking as well but these are the big ones, of course none of them are publicly available, but research papers are reputable. All of the ones mentioned should beat GPT-3 although GPT-3.5 (chatGPT) should be bit better and ability to search (Bing) should level the playing field even further, but Google's PaLM with search functionality should be clearly ahead. This is why people are excited about GPT-4, GPT-3 was way ahead of anyone else when it came out but others were able to catch up since, we'll see if GPT-4 will be another bing jump among LLMs.

Found relevant code at https://github.com/nvidia/megatron-lm + all code implementations here