tutel
Tutel MoE: An Optimized Mixture-of-Experts Implementation (by microsoft)
ModuleFormer
ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters. (by IBM)
| tutel | ModuleFormer | |
|---|---|---|
| 1 | 1 | |
| 658 | 216 | |
| 3.0% | 4.6% | |
| 6.5 | 5.7 | |
| 16 days ago | 25 days ago | |
| Python | Python | |
| MIT License | Apache License 2.0 |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tutel
Posts with mentions or reviews of tutel.
We have used some of these posts to build our list of alternatives
and similar projects.
-
Microsoft Research Introduces ‘Tutel’: A High-Performance MoE Library To Facilitate The Development Of Large-Scale DNN (Deep Neural Network) Models
You can read a short summary-based article written by Nitish (IIT Bhubaneshwar) here. The Github can be accessed here. If you are looking to read the full post on Microsoft, then you can read it here.
ModuleFormer
Posts with mentions or reviews of ModuleFormer.
We have used some of these posts to build our list of alternatives
and similar projects.
What are some alternatives?
When comparing tutel and ModuleFormer you can also consider the following projects:
hivemind - Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.
LMOps - General technology for enabling AI capabilities w/ LLMs and MLLMs
mixture-of-experts - PyTorch Re-Implementation of "The Sparsely-Gated Mixture-of-Experts Layer" by Noam Shazeer et al. https://arxiv.org/abs/1701.06538
StableLM - StableLM: Stability AI Language Models
transformers - 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
lingvo - Lingvo
lm-scorer - 📃Language Model based sentences scoring library