[D] Training 10x Larger Models and Accelerating Training with ZeRO-Offloading

This page summarizes the projects mentioned and recommended in the original post on reddit.com/r/MachineLearning

Our great sponsors
  • SonarQube - Static code analysis for 29 languages.
  • OPS - Build and Run Open Source Unikernels
  • Scout APM - Less time debugging, more time building
  • GitHub repo fairscale

    PyTorch extensions for high performance and large scale training.

    I created a feature request on the FairScale project so that we can track the progress on the integration: Support ZeRO-Offload · Issue #337 · facebookresearch/fairscale (github.com)

  • GitHub repo pytorch-lightning

    The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.

    I also asked for the respective support in PytorchLightning in this issue: Add deepspeed support · Issue #817 · PyTorchLightning/pytorch-lightning (github.com)

  • SonarQube

    Static code analysis for 29 languages.. Your projects are multi-language. So is SonarQube analysis. Find Bugs, Vulnerabilities, Security Hotspots, and Code Smells so you can release quality code every time. Get started analyzing your projects today for free.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts