Adan
dadaptation
| Adan | dadaptation | |
|---|---|---|
| 1 | 5 | |
| 727 | 483 | |
| 1.2% | 2.3% | |
| 4.6 | 5.6 | |
| about 1 month ago | 6 months ago | |
| Python | Python | |
| Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
Adan
-
Find Optimal Learning Rates for Stable Diffusion Fine-tunes (Link in Comments)
Adan https://github.com/sail-sg/Adan
dadaptation
-
D-Adaptation: Goodbye Learning Rate Headaches? (link in comments)
Just about a month ago, Facebook research published a paper called “Learning-Rate-Free Learning by D-Adaptation” (link) along with the code implementation (link). The paper is very technical but still worth the read regardless of your level. However, what it promises to deliver sounds very exciting and could save a lot of time spent on searching optimal parameters for different datasets and tasks:
-
Has anyone tried Facebook's learning-rate-free optimizer for Reinforcement Learning?
D-Adaption - https://github.com/facebookresearch/dadaptation
- Find Optimal Learning Rates for Stable Diffusion Fine-tunes (Link in Comments)
-
[R] Learning-Rate-Free Learning by D-Adaptation
Found relevant code at https://github.com/facebookresearch/dadaptation + all code implementations here
-
[D] "Deep Learning Tuning Playbook" (recently released by Google Brain people)
I tried out facebook's new learning-rate free version of Adam for a swin model I'm working on and it worked a little bit better than the best version of AdamW I found with a learning-rate sweep. https://github.com/facebookresearch/dadaptation
What are some alternatives?
Sophia - Effortless plugin and play Optimizer to cut model training costs by 50%. New optimizer that is 2x faster than Adam on LLMs.
tuning_playbook - A playbook for systematically maximizing the performance of deep learning models.
AdamP - AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights (ICLR 2021)
tuning_playbook - A playbook for systematically maximizing the performance of deep learning models.
Adan-pytorch - Implementation of the Adan (ADAptive Nesterov momentum algorithm) Optimizer in Pytorch
pytorch_resnet_cifar10 - Proper implementation of ResNet-s for CIFAR10/100 in pytorch that matches description of the original paper.
DeepFake-Detection - Towards deepfake detection that actually works
LaTeX-OCR - pix2tex: Using a ViT to convert images of equations into LaTeX code.