GPT-3 Is No Longer the Only Game in Town

• replika-research

  286 363 1.8 Jupyter Notebook

  Replika.ai Research Papers, Posters, Slides & Datasets

  

• hivemind

  40 1,842 5.4 Python

  Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.

  

The problem is that, currently, large ML models need to be trained on clusters of tightly-connected GPUs/accelerators. So it's kinda useless having a bunch of GPUs spread all over the world with huge latency and low bandwidth between them. That may change though - there are people working on it: https://github.com/learning-at-home/hivemind

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• mesh-transformer-jax

  52 6,213 0.0 Python

  Model parallel transformers in JAX and Haiku

I’m surprised that no one has answered for three hours!

The answer is at https://github.com/kingoflolz/mesh-transformer-jax

It has detailed comparisons and a full breakdown of the performance, courtesy of Eleuther.

Suggest a related project