How to Go Beyond Data Parallelism and Model Parallelism: Starting from GShard

This page summarizes the projects mentioned and recommended in the original post on news.ycombinator.com

Our great sponsors
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • WorkOS - The modern identity platform for B2B SaaS
  • SaaSHub - Software Alternatives and Reviews
  • oneflow

    OneFlow is a deep learning framework designed to be user-friendly, scalable and efficient.

    This article lists papers on GShard, presents background information and inspiration from the papers, and finally evaluates what else can be done to improve GShard from similar work that has been done in OneFlow.

    OneFlow Paper:https://arxiv.org/abs/2110.15032; Code:https://github.com/Oneflow-Inc/oneflow/

    The paper of Gshard contains two main parts of work, one on parallel APIs and one on Mixture of experts. The former part is more interesting and I will only discuss this part. The contribution on parallel APIs is outlined clearly in the abstract of the paper:

    GShard is a module composed of a set of lightweight annotation APIs and an extension to the XLA compiler.

    Gshard Paper: https://arxiv.org/pdf/2006.16668.pdf

  • InfluxDB

    Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts