Glossary term
Glossary term
Training and Fine-Tuning
A sophisticated gradient descent algorithm in which a learning step depends not only on the derivative in the current step, but also on the derivatives of the step(s) that immediately preceded it. Momentum involves computing an exponentially weighted moving average of the gradients over time, analogous to momentum in physics. Momentum sometimes prevents learning from getting stuck in local minima.
Created for this library
An ML team uses SGD with momentum 0.9 as the default optimizer for its production training pipelines.
A research team uses Nesterov momentum on selected experiments because it converges slightly faster than standard momentum on certain models.
A startup engineer increases momentum during the warm-up phase of training to accelerate convergence in the early epochs.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License