[D] PyTorch Distributed Training Libraries: What are the current options?

• accelerate

  18 6,909 9.7 Python

  🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

  

There's also huggingface accelerate to look at. It seems to require less changes to the codebase than the others (except you're using Lightning anyway). https://github.com/huggingface/accelerate

• horovod

  8 13,942 5.8 Python

  Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

  

Check out Horovod - https://github.com/horovod/horovod

• WorkOS

  workos.com sponsored

  The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

  

• stoke

  2 63 0.0 Python

  A lightweight wrapper for PyTorch that provides a simple declarative API for context switching between devices, distributed modes, mixed-precision, and PyTorch extensions. (by fidelity)

  

Suggest a related project