tutel
mixture-of-experts
tutel | mixture-of-experts | |
---|---|---|
1 | 2 | |
658 | 835 | |
3.0% | - | |
6.5 | 5.3 | |
16 days ago | 16 days ago | |
Python | Python | |
MIT License | GNU General Public License v3.0 only |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
tutel
-
Microsoft Research Introduces ‘Tutel’: A High-Performance MoE Library To Facilitate The Development Of Large-Scale DNN (Deep Neural Network) Models
You can read a short summary-based article written by Nitish (IIT Bhubaneshwar) here. The Github can be accessed here. If you are looking to read the full post on Microsoft, then you can read it here.
mixture-of-experts
- [Rumor] Potential GPT-4 architecture description
-
Local and Global loss
I have a requirement of training pipeline similar to Mixture of Experts (https://github.com/davidmrau/mixture-of-experts/blob/master/moe.py) but I want to train the Experts on a local loss for 1 epoch before predicting outputs from them (which would then be concatenated for the global loss of MoE). Can anyone suggest what’s the best way to set up this training pipeline?
What are some alternatives?
hivemind - Decentralized deep learning in PyTorch. Built to train models on thousands of volunteers across the world.
pytorch-tutorial - PyTorch Tutorial for Deep Learning Researchers
transformers - 🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
ModuleFormer - ModuleFormer is a MoE-based architecture that includes two different types of experts: stick-breaking attention heads and feedforward experts. We released a collection of ModuleFormer-based Language Models (MoLM) ranging in scale from 4 billion to 8 billion parameters.
mmdetection - OpenMMLab Detection Toolbox and Benchmark
yolov5 - YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
Real-Time-Voice-Cloning - Clone a voice in 5 seconds to generate arbitrary speech in real-time