Implementation of mixture of experts language model in a single file of PyTorch

• makeMoE

  3 513 9.0 Jupyter Notebook

  From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)

• moegpt

  1 1 4.9 Python

  Mixture of Experts (MoE) implementation for minGPT

Similar MoE implementation was on GitHub for a while, since Jan 2024

https://github.com/zxaall/moegpt

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• MoE

  1 1 5.5 Python

  some mixture of experts architecture implementations (by swiss-ai)

  

very cool work! we did something similar in the context of the Swiss AI initiative (https://www.swiss-ai.org/) here: https://github.com/swiss-ai/MoE. The implementation is as simple and fast as nanoGPT and works with our modular llm-baselines codebase (https://github.com/epfml/llm-baselines) for experimenting with transformers and different datasets :)

• llm-baselines

  1 60 6.9 Python
  

very cool work! we did something similar in the context of the Swiss AI initiative (https://www.swiss-ai.org/) here: https://github.com/swiss-ai/MoE. The implementation is as simple and fast as nanoGPT and works with our modular llm-baselines codebase (https://github.com/epfml/llm-baselines) for experimenting with transformers and different datasets :)

Suggest a related project