st-moe-pytorch
OpenMoE
st-moe-pytorch | OpenMoE | |
---|---|---|
1 | 8 | |
231 | 1,233 | |
- | - | |
7.8 | 8.6 | |
3 months ago | 2 months ago | |
Python | Python | |
MIT License | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
st-moe-pytorch
OpenMoE
-
Mixtral: A Promising Model with Unforeseen Challenges
Switch Transformer aside, it's not very open and OpenMoE is months old.
- will the point meet in 2024?
-
Ask HN: Why is GPT4 better than the other major LLMs?
What about this one? https://github.com/XueFuzhao/OpenMoE
-
Partial Outage Across ChatGPT and API
https://github.com/XueFuzhao/OpenMoE
Check out this open source Mixture of Experts research. Could help a lot with performance of open source models.
-
OpenAI is too cheap to beat
I think the weird thing about this is that it's completely true right now but in X months it may be totally outdated advice.
For example, efforts like OpenMOE https://github.com/XueFuzhao/OpenMoE or similar will probably eventually lead to very competitive performance and cost-effectiveness for open source models. At least in terms of competing with GPT-3.5 for many applications.
Also see https://laion.ai/
I also believe that within say 1-3 years there will be a different type of training approach that does not require such large datasets or manual human feedback.
-
Mixtures of Experts
Google have released the models and code for the Switch Transformer from Fedus et al. (2021) under the Apache 2.0 licence. [0]
There's also OpenMoE - an open-source effort to train a mixture of experts model. Currently they've released a model with 8 billion parameters. [1]
[0] https://github.com/google-research/t5x/blob/main/docs/models...
[1] https://github.com/XueFuzhao/OpenMoE
- A Hackers' Guide to Language Models [video]
- OpenMoE – A family of open-sourced Mixture-of-Experts (MoE) LLMs
What are some alternatives?
spaCy - 💫 Industrial-strength Natural Language Processing (NLP) in Python
lm-hackers - Hackers' Guide to Language Models
t5x
Laminoid - An ML instance manager