Fork of kingoflolz/mesh-transformer-jax with memory usage optimizations and support for GPT-Neo, GPT-NeoX, BLOOM, OPT and fairseq dense LM. Primarily used by KoboldAI and mtj-softtuner.
Why do you think that https://github.com/googlecolab/colabtools is a good alternative to mesh-transformer-jax