flaxOptimizers
lion-pytorch
flaxOptimizers | lion-pytorch | |
---|---|---|
1 | 3 | |
28 | 1,920 | |
- | - | |
0.0 | 3.8 | |
over 2 years ago | about 1 month ago | |
Python | Python | |
Apache License 2.0 | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
flaxOptimizers
-
[P] Implementation of MADGRAD optimization algorithm for Tensorflow
For those who are interested, I have a Flax implementation of MADGRAD in flaxOptimizers (here). The optimizer solid and a refreshing departure from Adam-derived optimizers. One big caveat, however, is that you will need to tune your hyperparameters as they are likely to be orders of magnitude different from Adam's value.
lion-pytorch
-
Applying All Recent Innovations To Train a Code Model
Various people are trying LiON on their projects, with varying degrees of success. A good starting point to look around is the lion-pytorch on github from Phil Wang aka lucidrains (thank you man!).
- AMD RoCM Dockerfiles
-
[D] Lion , An Optimizer That Outperforms Adam - Symbolic Discovery of Optimization Algorithms
Code Implementation: https://github.com/lucidrains/lion-pytorch
What are some alternatives?
ML-Optimizers-JAX - Toy implementations of some popular ML optimizers using Python/JAX
llm-foundry - LLM training code for Databricks foundation models
opytimizer - 🐦 Opytimizer is a Python library consisting of meta-heuristic optimization algorithms.
dnn_from_scratch - A high level deep learning library for Convolutional Neural Networks,GANs and more, made from scratch(numpy/cupy implementation).
pytorch-lightning - Pretrain, finetune and deploy AI models on multiple GPUs, TPUs with zero code changes.
Gradient-Centralization-TensorFlow - Instantly improve your training performance of TensorFlow models with just 2 lines of code!
RoCMyDocker - A collection of Dockerfiles tailored for Deep Learning and other GPU-accelerated applications on AMD Radeon GPUs using ROCm
spaCy - 💫 Industrial-strength Natural Language Processing (NLP) in Python