mesh-transformer-jax
Model parallel transformers in JAX and Haiku (by AeroScripts)
mesh-transformer-jax
Model parallel transformers in JAX and Haiku (by kingoflolz)
| mesh-transformer-jax | mesh-transformer-jax | |
|---|---|---|
| 1 | 52 | |
| 0 | 6,213 | |
| - | - | |
| 0.0 | 0.0 | |
| almost 3 years ago | over 1 year ago | |
| Jupyter Notebook | Python | |
| Apache License 2.0 | Apache License 2.0 |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
mesh-transformer-jax
Posts with mentions or reviews of mesh-transformer-jax.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-06-08.
-
GPT-J-6B, a 6B Parameter Text Generation Model
I'm running it quite comfortably on my 3090, although it's a really snug fit for the VRAM, and that's with a number of fixes to significantly reduce its memory use from https://github.com/AeroScripts/mesh-transformer-jax .
mesh-transformer-jax
Posts with mentions or reviews of mesh-transformer-jax.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2024-02-12.
-
Large Language Models: Compairing Gen2/Gen3 Models (GPT-3, GPT-J, MT5 and More)
GPT-J is a LLM case study with two goals: Training a LLM with a data source containing unique material, and using the training frameworkMesh Transformer JAX to achieve a high training efficiency through parallelization. There is no research paper about GPT-J, but on its GitHub pages, the model, different checkpoints, and the complete source code for training is given.
-
[R] Parallel Attention and Feed-Forward Net Design for Pre-training and Inference on Transformers
This idea has already been proposed in ViT-22B and GPT-J-6B.
- Show HN: Finetune LLaMA-7B on commodity GPUs using your own text
-
[D] An Instruct Version Of GPT-J Using Stanford Alpaca's Dataset
Sure. Here's the repo I used for the fine-tuning: https://github.com/kingoflolz/mesh-transformer-jax. I used 5 epochs, and appart from that I kept the default parameters in the repo.
- Boss wants me to use ChatGPT for work, but I refuse to input my personal phone number. Any advice?
-
Let's build GPT: from scratch, in code, spelled out by Andrej Karpathy
You can skip to step 4 using something like GPT-J as far as I understand: https://github.com/kingoflolz/mesh-transformer-jax#links
The pretrained model is already available.
-
Best coding model?
The Github repo suggests it's possible you can change the number of checkpoints to make it run on a GPU.
- Ask HN: What language models can I fine-tune at home?
-
selfhosted/ open-source ChatGPT alternative?
GPT-J, which uses mesh-transformer-jax: https://github.com/kingoflolz/mesh-transformer-jax
-
GPT-J, an open-source alternative to GPT-3
They hinted at it in the screenshot, but the goods are linked from the https://6b.eleuther.ai page: https://github.com/kingoflolz/mesh-transformer-jax#gpt-j-6b (Apache 2)
What are some alternatives?
When comparing mesh-transformer-jax and mesh-transformer-jax you can also consider the following projects:
jax - Composable transformations of Python+NumPy programs: differentiate, vectorize, JIT to GPU/TPU, and more
DeepSpeed - DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
the-pile
tensorflow - An Open Source Machine Learning Framework for Everyone
gpt-neo - An implementation of model parallel GPT-2 and GPT-3-style models using the mesh-tensorflow library.
KoboldAI-Client
alpaca-lora - Instruct-tune LLaMA on consumer hardware
Finetune_LLMs - Repo for fine-tuning Casual LLMs
gpt-2 - Code for the paper "Language Models are Unsupervised Multitask Learners"
cedille-ai - ✒️ Cedille is a large French language model (6B), released under an open-source license
replika-research - Replika.ai Research Papers, Posters, Slides & Datasets
mesh-transformer-jax vs jax
mesh-transformer-jax vs DeepSpeed
mesh-transformer-jax vs the-pile
mesh-transformer-jax vs tensorflow
mesh-transformer-jax vs gpt-neo
mesh-transformer-jax vs jax
mesh-transformer-jax vs KoboldAI-Client
mesh-transformer-jax vs alpaca-lora
mesh-transformer-jax vs Finetune_LLMs
mesh-transformer-jax vs gpt-2
mesh-transformer-jax vs cedille-ai
mesh-transformer-jax vs replika-research