[D] PyTorch Distributed Training Libraries: What are the current options?

1. accelerate

   1 25 9,072 9.5 Python

   🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support

   

   There's also huggingface accelerate to look at. It seems to require less changes to the codebase than the others (except you're using Lightning anyway). https://github.com/huggingface/accelerate

2. InfluxDB

   www.influxdata.com featured

   InfluxDB – Built for High-Performance Time Series Workloads. InfluxDB 3 OSS is now GA. Transform, enrich, and act on time series data directly in the database. Automate critical tasks and eliminate the need to move data externally. Download now.

   

3. horovod

   2 8 14,577 4.3 Python

   Distributed training framework for TensorFlow, Keras, PyTorch, and Apache MXNet.

   

   Check out Horovod - https://github.com/horovod/horovod

4. stoke

   3 2 66 0.0 Python

   Discontinued A lightweight wrapper for PyTorch that provides a simple declarative API for context switching between devices, distributed modes, mixed-precision, and PyTorch extensions. (by fidelity)

   

Suggest a related project