SGD-OGR-Hessian-estimator
backpack
| SGD-OGR-Hessian-estimator | backpack | |
|---|---|---|
| 8 | 2 | |
| 10 | 541 | |
| - | - | |
| 4.9 | 1.9 | |
| 12 months ago | 2 days ago | |
| Mathematica | Python | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
SGD-OGR-Hessian-estimator
-
Can you realistically write own neural network training optimizer in Mathematica?
I have developed new approach for optimizer (sources, article: https://github.com/JarekDuda/SGD-OGR-Hessian-estimator ) - estimating Hessian from online linear regression of gradients, in evolving locally interesting subspace.
- [R] SGD augmented with 2nd order information from seen sequence of gradients?
-
Improving gradient descent convergence e.g. based on local trend of gradients?
Regarding comparison with momentum, here is using the largest learning rate (without escaping to infinity) for the same scenario - leading to ~50x worse values after 30 steps: https://github.com/JarekDuda/SGD-OGR-Hessian-estimator/raw/main/momentum.png
-
SGD augmented with 2nd order information from seen sequence of gradients?
Here Hessian is estimated from linear regression of seen gradients, source: https://github.com/JarekDuda/SGD-OGR-Hessian-estimator
- [R] SGD augmented with 2nd order information from seen sequence of gradients - for nasty Beale function starts approaching in ~10 steps
- SGD augmented with 2nd order information from seen sequence of gradients - for nasty Beale function starts approaching in ~10 steps
backpack
What are some alternatives?
higher - higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.
cleverhans - An adversarial example library for constructing attacks, building defenses, and benchmarking both
TorchDrift - Drift Detection for your PyTorch Models
explainerdashboard - Quickly build Explainable AI dashboards that show the inner workings of so-called "blackbox" machine learning models.
uncertainty-toolbox - Uncertainty Toolbox: a Python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization
pnotify - Beautiful JavaScript notifications with Web Notifications support.
pytea - PyTea: PyTorch Tensor shape error analyzer
AIX360 - Interpretability and explainability of data and machine learning models
delve - PyTorch model training and layer saturation monitor
cockpit - Cockpit: A Practical Debugging Tool for Training Deep Neural Networks
yellowbrick - Visual analysis and diagnostic tools to facilitate machine learning model selection.
vivit - [TMLR 2022] Curvature access through the generalized Gauss-Newton's low-rank structure: Eigenvalues, eigenvectors, directional derivatives & Newton steps