Mup Alternatives

mup reviews and mentions

• Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
  1 project | news.ycombinator.com | 16 Jun 2025
• GPT-5 is behind schedule
  5 projects | news.ycombinator.com | 22 Dec 2024

  Because of mup [0] and scaling laws, you can test ideas empirically on smaller models, with some confidence they will transfer to the larger model.

  [0] https://arxiv.org/abs/2203.03466

• Announcing xAI July 12th 2023
  3 projects | /r/xdotai | 13 Jul 2023

  Our team is led by Elon Musk, CEO of Tesla and SpaceX. We have previously worked at DeepMind, OpenAI, Google Research, Microsoft Research, Tesla, and the University of Toronto. Collectively we contributed some of the most widely used methods in the field, in particular the Adam optimizer, Batch Normalization, Layer Normalization, and the discovery of adversarial examples. We further introduced innovative techniques and analyses such as Transformer-XL, Autoformalization, the Memorizing Transformer, Batch Size Scaling, and μTransfer. We have worked on and led the development of some of the largest breakthroughs in the field including AlphaStar, AlphaCode, Inception, Minerva, GPT-3.5, and GPT-4.

• Bard is getting better at logic and reasoning
  1 project | news.ycombinator.com | 7 Jun 2023

  I believe tuning hyper parameters well without a lot of waste for the largest models was only figured out by Greg Yang/Microsoft Research around 2022 (cited in GPT-4 paper):

  https://arxiv.org/abs/2203.03466

  Also part of how they predicted the loss ahead of time so well.

• Cerebras Open Sources Seven GPT models and Introduces New Scaling Law
  3 projects | /r/mlscaling | 28 Mar 2023

  This is the first time I have seen muP applied by the third party. See Cerebras Model Zoo, where muP models have scale-invariant constant LR.

• OpenAI’s policies hinder reproducible research on language models
  2 projects | news.ycombinator.com | 23 Mar 2023

  I guess, but its actually not simple to do that, in my experience. There’s another paper on that: https://arxiv.org/abs/2203.03466

  Why isn’t chinchilla running google AI chat or whatever then?

• [D] Anyone else witnessing a panic inside NLP orgs of big tech companies?
  3 projects | /r/MachineLearning | 16 Mar 2023

  Well, but it isn't like this kind of research is new. Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer (2022) tuned hyperparameters in 40M model, transferred it to 6.7B model, and beat OpenAI's 6.7B run. It is likely what OpenAI did is perfecting this kind of research. I note that four authors of that paper (Igor Babuschkin, Szymon Sidor, David Farhi, Jakub Pachocki) are credited for pretraining optimization & architecture at https://openai.com/contributions/gpt-4.

• [R] Greg Yang's work on a rigorous mathematical theory for neural networks
  4 projects | /r/MachineLearning | 7 Jan 2023

  Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes: https://arxiv.org/abs/1910.12478 Tensor Programs II: Neural Tangent Kernel for Any Architecture: https://arxiv.org/abs/2006.14548 Tensor Programs III: Neural Matrix Laws: https://arxiv.org/abs/2009.10685 Tensor Programs IV: Feature Learning in Infinite-Width Neural Networks: https://proceedings.mlr.press/v139/yang21c.html Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer: https://arxiv.org/abs/2203.03466

• [D] How does one choose a learning rate schedule for models that take days or weeks to train?
  2 projects | /r/MachineLearning | 15 Sep 2022
• How to do meaningful work as an independent researcher? [Discussion]
  2 projects | /r/MachineLearning | 28 Apr 2022
• A note from our sponsor - SaaSHub
  www.saashub.com | 16 Aug 2026

  SaaSHub helps you find the best software and product alternatives Learn more →