backpack
SGD-OGR-Hessian-estimator
Our great sponsors
| backpack | SGD-OGR-Hessian-estimator | |
|---|---|---|
| 2 | 8 | |
| 541 | 10 | |
| - | - | |
| 2.8 | 4.9 | |
| about 2 months ago | 12 months ago | |
| Python | Mathematica | |
| MIT License | MIT License |
Stars - the number of stars that a project has on GitHub. Growth - month over month growth in stars.
Activity is a relative number indicating how actively a project is being developed. Recent commits have higher weight than older ones.
For example, an activity of 9.0 indicates that a project is amongst the top 10% of the most actively developed projects that we are tracking.
backpack
SGD-OGR-Hessian-estimator
-
Can you realistically write own neural network training optimizer in Mathematica?
I have developed new approach for optimizer (sources, article: https://github.com/JarekDuda/SGD-OGR-Hessian-estimator ) - estimating Hessian from online linear regression of gradients, in evolving locally interesting subspace.
- [R] SGD augmented with 2nd order information from seen sequence of gradients?
-
Improving gradient descent convergence e.g. based on local trend of gradients?
Regarding comparison with momentum, here is using the largest learning rate (without escaping to infinity) for the same scenario - leading to ~50x worse values after 30 steps: https://github.com/JarekDuda/SGD-OGR-Hessian-estimator/raw/main/momentum.png
-
SGD augmented with 2nd order information from seen sequence of gradients?
Here Hessian is estimated from linear regression of seen gradients, source: https://github.com/JarekDuda/SGD-OGR-Hessian-estimator
- [R] SGD augmented with 2nd order information from seen sequence of gradients - for nasty Beale function starts approaching in ~10 steps
- SGD augmented with 2nd order information from seen sequence of gradients - for nasty Beale function starts approaching in ~10 steps
What are some alternatives?
cleverhans - An adversarial example library for constructing attacks, building defenses, and benchmarking both
higher - higher is a pytorch library allowing users to obtain higher order gradients over losses spanning training loops rather than individual training steps.
TorchDrift - Drift Detection for your PyTorch Models
explainerdashboard - Quickly build Explainable AI dashboards that show the inner workings of so-called "blackbox" machine learning models.
uncertainty-toolbox - Uncertainty Toolbox: a Python toolbox for predictive uncertainty quantification, calibration, metrics, and visualization
pnotify - Beautiful JavaScript notifications with Web Notifications support.
pytea - PyTea: PyTorch Tensor shape error analyzer
AIX360 - Interpretability and explainability of data and machine learning models
delve - PyTorch model training and layer saturation monitor
cockpit - Cockpit: A Practical Debugging Tool for Training Deep Neural Networks
yellowbrick - Visual analysis and diagnostic tools to facilitate machine learning model selection.
vivit - [TMLR 2022] Curvature access through the generalized Gauss-Newton's low-rank structure: Eigenvalues, eigenvectors, directional derivatives & Newton steps