cedille-ai VS mesh-transformer-jax

GPT-3’s release has inspired a gold rush, with over 30 new large language models trained since May/2020, especially through North America and China, but also in places like Israel, Germany, Switzerland, and Abu Dhabi.

Le repo sur GitHub : https://github.com/coteries/cedille-ai

We are excited to announce Cedille, the largest language model for French (6b parameters).

Demo: https://cedille.ai

Language models are general purpose AI systems that are able to solve a range of tasks by simply being prompted for it. It can be used for example to summarize text, do translations, or for idea generation & overcoming writer's block.

You may know GPT-3, the humongous model from OpenAI. Cedille is a similar model targeting the French demographic - but smaller, as we don’t yet have $1b in the bank like they do. Although GPT-3 supports multiple languages including French, our model is competitive with GPT-3 on a range of French tasks! Plus, of course we’re open source while they keep their model closed and heavily restrict access to it.

You can try it out right away from our playground: https://app.cedille.ai

We are proponents of “open AI” and as such have released a checkpoint for the world to use (MIT license): https://github.com/coteries/cedille-ai

One of the problems with large language models is the potentially toxic, sexist or in other ways unpleasant output. We tried our best to avoid this issue by doing extensive dataset filtering. As a result, our benchmark indicates that Cedille is indeed less toxic than GPT-3.

We are proponents of “open AI” and as such have released a checkpoint for the world to use (MIT license) : https://github.com/coteries/cedille-ai

GPT-J is a LLM case study with two goals: Training a LLM with a data source containing unique material, and using the training frameworkMesh Transformer JAX to achieve a high training efficiency through parallelization. There is no research paper about GPT-J, but on its GitHub pages, the model, different checkpoints, and the complete source code for training is given.

This idea has already been proposed in ViT-22B and GPT-J-6B.

Sure. Here's the repo I used for the fine-tuning: https://github.com/kingoflolz/mesh-transformer-jax. I used 5 epochs, and appart from that I kept the default parameters in the repo.

You can skip to step 4 using something like GPT-J as far as I understand: https://github.com/kingoflolz/mesh-transformer-jax#links

The pretrained model is already available.

The Github repo suggests it's possible you can change the number of checkpoints to make it run on a GPU.

GPT-J, which uses mesh-transformer-jax: https://github.com/kingoflolz/mesh-transformer-jax

They hinted at it in the screenshot, but the goods are linked from the https://6b.eleuther.ai page: https://github.com/kingoflolz/mesh-transformer-jax#gpt-j-6b (Apache 2)