Glossary term
Glossary term
Training and Fine-Tuning
The tendency for gradients in deep neural networks (especially recurrent neural networks) to become surprisingly steep (high). Steep gradients often cause very large updates to the weights of each node in a deep neural network.
Models suffering from the exploding gradient problem become difficult or impossible to train. Gradient clipping can mitigate this problem.
Compare to vanishing gradient problem.
Created for this library
An NLP team encounters the exploding gradient problem when training a long-sequence RNN and adds gradient clipping to stabilize training.
A speech recognition team adds gradient clipping during recurrent model training after running into the exploding gradient problem on long audio sequences.
An ML platform team standardizes gradient clipping defaults across all jobs to prevent the exploding gradient problem from killing long-running training runs.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License