DemonRangerOptimizer
ML-Optimizers-JAX
| DemonRangerOptimizer | ML-Optimizers-JAX | |
|---|---|---|
| 1 | 1 | |
| 23 | 40 | |
| - | - | |
| 0.0 | 4.5 | |
| over 3 years ago | almost 3 years ago | |
| Python | Python | |
| - | - |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
DemonRangerOptimizer
-
[R] AdasOptimizer Update: Cifar-100+MobileNetV2 Adas generalizes with Adas 15% better and 9x faster than Adam
The results are interesting, but in terms of novelty of the main theory - isn't it almost identical to Baydin et al.? https://arxiv.org/pdf/1703.04782.pdf It seems the difference may be in some implementation details, like using a running average for the past gradient. If it's useful, I implemented a bunch of optimizers with options to synergize different techniques (https://github.com/JRC1995/DemonRangerOptimizer) including hypergradient updates for stuffs (and taking into account decorrelated weight decay and per-parameter lrs for hypergradient lr) when I was bored before practically abandoning it all together. I didn't really run any experiments with it though, but some people tried although they may not have got any particularly striking results.
ML-Optimizers-JAX
-
ML Optimizers from scratch using JAX
Github link (includes a link to a Kaggle notebook to run it directly) - shreyansh26/ML-Optimizers-JAX
What are some alternatives?
pytorch-optimizer - torch-optimizer -- collection of optimizers for Pytorch
RAdam - On the Variance of the Adaptive Learning Rate and Beyond
AdasOptimizer - ADAS is short for Adaptive Step Size, it's an optimizer that unlike other optimizers that just normalize the derivative, it fine-tunes the step size, truly making step size scheduling obsolete, achieving state-of-the-art training performance
dm-haiku - JAX-based neural network library
imagenette - A smaller subset of 10 easily classified classes from Imagenet, and a little more French
trax - Trax — Deep Learning with Clear Code and Speed
Gradient-Centralization-TensorFlow - Instantly improve your training performance of TensorFlow models with just 2 lines of code!
dnn_from_scratch - A high level deep learning library for Convolutional Neural Networks,GANs and more, made from scratch(numpy/cupy implementation).
flaxOptimizers - A collection of optimizers, some arcane others well known, for Flax.
yaglm - A python package for penalized generalized linear models that supports fitting and model selection for structured, adaptive and non-convex penalties.