deepnet
RAdam
deepnet | RAdam | |
---|---|---|
1 | 4 | |
319 | 2,520 | |
- | - | |
0.0 | 0.0 | |
almost 2 years ago | almost 3 years ago | |
Python | Python | |
MIT License | Apache License 2.0 |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
deepnet
-
Neural Network from Scratch
For those interested in simple neural networks to CNN and RNNs implemented with just Numpy (including backprop):
https://github.com/parasdahal/deepnet
RAdam
-
[D] Why does a sudden increase in accuracy at a specific epoch in these model
Code for https://arxiv.org/abs/1908.03265 found: https://github.com/LiyuanLucasLiu/RAdam
-
[D] How to pick a learning rate scheduler?
common practice is to include some type of annealing (cosine, linear, etc.), which makes intuitive sense. for adam/adamw, it's generally a good idea to include a warmup in the lr schedule, as the gradient distribution without the warmup can be distorted, leading to the optimizer being trapped in a bad local min. see this paper. there are also introduced in this paper and subsequent works (radam, ranger, and variants) that don't require a warmup stage to stabilize the gradients. i would say in general, if you're using adam/adamw, include a warmup and some annealing, either linear or cosine. if you're using radam/ranger/variants, you can skip the warmup. how many steps to use for warmup/annealing are probably problem specific, and require some hyperparam tuning to get optimimal results
- Why is my loss choppy?
What are some alternatives?
micrograd - A tiny scalar-valued autograd engine and a neural net library on top of it with PyTorch-like API
ML-Optimizers-JAX - Toy implementations of some popular ML optimizers using Python/JAX
ML-From-Scratch - Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.
AdaBound - An optimizer that trains as fast as Adam and as good as SGD.
NNfSiX - Neural Networks from Scratch in various programming languages
pytorch_warmup - Learning Rate Warmup in PyTorch
MachineLearning - From linear regression towards neural networks...
pytorch-optimizer - torch-optimizer -- collection of optimizers for Pytorch
machine.academy - Neural Network training library in C++ and C# with GPU acceleration
DemonRangerOptimizer - Quasi Hyperbolic Rectified DEMON Adam/Amsgrad with AdaMod, Gradient Centralization, Lookahead, iterative averaging and decorrelated Weight Decay