[R] You can't train GPT-3 on a single GPU, but you *can* tune its hyperparameters on one

• gpt-neox

  52 6,569 8.9 Python

  An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library.

  

Here is the codebase that has trained the largest publicly available GPT-3-style model. Here is the codebase that that has trained the second largest publicly available GPT-3-style model. Here is another codebase that has trained even larger models than GPT-3 which is public, though the largest models it has trained are not.

• fairseq

  89 29,205 6.6 Python

  Facebook AI Research Sequence-to-Sequence Toolkit written in Python.

  

Here is the codebase that has trained the largest publicly available GPT-3-style model. Here is the codebase that that has trained the second largest publicly available GPT-3-style model. Here is another codebase that has trained even larger models than GPT-3 which is public, though the largest models it has trained are not.

• InfluxDB

  www.influxdata.com sponsored

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• Megatron-DeepSpeed

  1 1,590 9.0 Python

  Ongoing research training transformer language models at scale, including: BERT & GPT-2 (by microsoft)

  

Here is the codebase that has trained the largest publicly available GPT-3-style model. Here is the codebase that that has trained the second largest publicly available GPT-3-style model. Here is another codebase that has trained even larger models than GPT-3 which is public, though the largest models it has trained are not.

Suggest a related project