sam
SAM: Sharpness-Aware Minimization (PyTorch) (by davda54)
simple-sam
Sharpness-Aware Minimization for Efficiently Improving Generalization (by Jannoshh)
| sam | simple-sam | |
|---|---|---|
| 3 | 1 | |
| 1,655 | 40 | |
| - | - | |
| 0.0 | 0.0 | |
| 3 months ago | about 2 years ago | |
| Python | Python | |
| MIT License | MIT License |
The number of mentions indicates the total number of mentions that we've tracked plus the number of user suggested alternatives.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
sam
Posts with mentions or reviews of sam.
We have used some of these posts to build our list of alternatives
and similar projects.
-
What is the correct way to sum loss into a total loss and then to backprop?
Which from here I understand that I shouldn't use the same loss variable for both forward passes but I'm not sure how else to do this. I thought that I could maybe create a variable called total_loss and add the loss to it and then after the iterations to backprop over it, but I'm not sure if that's the correct approach.
-
[R] Sharpness-Aware Minimization for Efficiently Improving Generalization
They reached sota on a few tasks. Do you really believe that the entire community missed the magic hyperparameters of batch size 128 and Adam to beat SOTA? I think getting SOTA really solidifies the approach, albeit the 2x speed cost seems heavy. As for implementation, it looks fairly trivial to adapt to all optimizers, at least from this random github https://github.com/davda54/sam
-
Help me implement this paper expanding on Google's SAM optimizer
Here is the code for SAM. SAM isn't too complicated. There are two forward and backward passes, a gradient accent after the first one, and the gradient decent after the second. The gradient accent is to get the noised SAM model which is calculated as for each p in param group add epsilon, which is rho * (p.grad / grad_norm), with rho being SAM's only hyperparameter.
simple-sam
Posts with mentions or reviews of simple-sam.
We have used some of these posts to build our list of alternatives
and similar projects. The last one was on 2021-06-05.
-
[R] When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations
Which implementation(s) have you been using? I've been eyeing this one for use in a Keras project of mine, so I'm curious what others are using.
What are some alternatives?
When comparing sam and simple-sam you can also consider the following projects:
pytorch-optimizer - torch-optimizer -- collection of optimizers for Pytorch
southpaw - Python Fanduel API (2023) - Lineup Automation
AdamP - AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (ICLR 2021)
PHP Documentor 3 - Documentation Generator for PHP
RAdam - On the Variance of the Adaptive Learning Rate and Beyond