[R] AdasOptimizer Update: Cifar-100+MobileNetV2 Adas generalizes with Adas 15% better and 9x faster than Adam

• AdasOptimizer

  2 85 5.8 C++

  ADAS is short for Adaptive Step Size, it's an optimizer that unlike other optimizers that just normalize the derivative, it fine-tunes the step size, truly making step size scheduling obsolete, achieving state-of-the-art training performance

I think too, I was comfortable with posting this because it was the same code for each optimizer. https://github.com/YanaiEliyahu/AdasOptimizer/blob/master/misc/cifar-100-mobilenetv2/model_with_training.py.txt if you care to find what I did wrong, go for it.

• imagenette

  9 877 0.0 Jupyter Notebook

  A smaller subset of 10 easily classified classes from Imagenet, and a little more French

  

You don't need Imagenet to verify it really works or not, Checkout https://github.com/fastai/imagenette the fastai folks have a small subset of Imagenet, which has 3 types of the dataset, test on them. If AdasOptimizer really works it you should be able to beat their results, or at least see where it stands.

• InfluxDB

  www.influxdata.com featured

  Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.

  

• DemonRangerOptimizer

  1 23 0.0 Python

  Quasi Hyperbolic Rectified DEMON Adam/Amsgrad with AdaMod, Gradient Centralization, Lookahead, iterative averaging and decorrelated Weight Decay

The results are interesting, but in terms of novelty of the main theory - isn't it almost identical to Baydin et al.? https://arxiv.org/pdf/1703.04782.pdf It seems the difference may be in some implementation details, like using a running average for the past gradient. If it's useful, I implemented a bunch of optimizers with options to synergize different techniques (https://github.com/JRC1995/DemonRangerOptimizer) including hypergradient updates for stuffs (and taking into account decorrelated weight decay and per-parameter lrs for hypergradient lr) when I was bored before practically abandoning it all together. I didn't really run any experiments with it though, but some people tried although they may not have got any particularly striking results.

• pytorch-optimizer

  3 2,946 3.1 Python

  torch-optimizer -- collection of optimizers for Pytorch

Suggest a related project