How to Train Large Models on Many GPUs?

• DeepSpeed

  16 32,447 9.8 Python

  DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

  

  DeepSpeed [1] is amazing tool to enable the different kind of parallelisms and optimizations on your model. I would definitely not recommend reimplementing everything yourself.

  Probably FairScale [2] too, but never tried it myself.

  [1]: https://github.com/microsoft/DeepSpeed

  [2]: https://github.com/facebookresearch/fairscale

• fairscale

  0 2,852 2.8 Python

  PyTorch extensions for high performance and large scale training.

  

  DeepSpeed [1] is amazing tool to enable the different kind of parallelisms and optimizations on your model. I would definitely not recommend reimplementing everything yourself.

  Probably FairScale [2] too, but never tried it myself.

  [1]: https://github.com/microsoft/DeepSpeed

  [2]: https://github.com/facebookresearch/fairscale

• WorkOS

  workos.com

  sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

Suggest a related project