Our great sponsors
-
We mostly stuck to the finetuning recommendations provided by GPT-J: https://github.com/kingoflolz/mesh-transformer-jax/blob/master/howto_finetune.md
-
We are proponents of “open AI” and as such have released a checkpoint for the world to use (MIT license) : https://github.com/coteries/cedille-ai
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
-
Another aspect we had fun with is dataset filtering. We have run the whole C4 French dataset through the Detoxify classifier to clean it up 🤬
-
We tried to overcome these problems to the best of our ability - Happy to answer if you have more specific questions! FYI: We used/adapted EleutherAI's eval harness (https://github.com/EleutherAI/lm-evaluation-harness) for most of this work.
Related posts
- Cedille, the largest French language model, open source with a freely accessible playground
- Integrating Hugging Face Transformers & DagsHub
- How to Train Large Models on Many GPUs?
- BetterTransformer: PyTorch-native free-lunch speedups for Transformer-based models
- FauxPilot – an open-source GitHub Copilot server