Distributed training with Horovod/MPI

• sagemaker-training-toolkit

  1 468 6.3 Python

  Train machine learning models within a 🐳 Docker container using 🧠 Amazon SageMaker.

  

I'm using sagemaker-training-toolkit to attempt hyperparameter optimization and trying to take advantage of all the cores on each machine using their MPI options (which uses Horovod with MPI to my understanding). I'm pretty new to this space and can't find anything that describes in somewhat lay-terms how training works in this distributed model. With AllReduce, how often does the reduce happen? I'm trying to figure out if all training threads are training a shared model such that every thread is training on the "latest" version of the model.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

Suggest a related project