Improving gradient descent convergence e.g. based on local trend of gradients?

This page summarizes the projects mentioned and recommended in the original post on /r/math

Our great sponsors
  • WorkOS - The modern identity platform for B2B SaaS
  • InfluxDB - Power Real-Time Data Analytics at Scale
  • SaaSHub - Software Alternatives and Reviews
  • SGD-OGR-Hessian-estimator

    SGD (stochastic gradient descent) with OGR - online gradient regression Hessian estimator

  • So it directly estimates learning rates based on local trend of gradients, here is analogous scenario for standard momentum methods with maximal fixed learning rate: much smaller steps, after 30 steps ~50x worse values: https://github.com/JarekDuda/SGD-OGR-Hessian-estimator/raw/main/momentum.png

  • WorkOS

    The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.

    WorkOS logo
NOTE: The number of mentions on this list indicates mentions on common posts plus user suggested alternatives. Hence, a higher number means a more popular project.

Suggest a related project

Related posts