Glossary term
Glossary term
Training and Fine-Tuning
The tendency for the gradients of early hidden layers of some deep neural networks to become surprisingly flat (low). Increasingly lower gradients result in increasingly smaller changes to the weights on nodes in a deep neural network, leading to little or no learning. Models suffering from the vanishing gradient problem become difficult or impossible to train. Long Short-Term Memory cells address this issue.
Compare to exploding gradient problem.
Created for this library
A research team uses residual connections in deep networks to mitigate the vanishing gradient problem.
An NLP team uses LSTMs rather than vanilla RNNs to mitigate the vanishing gradient problem on long sequences.
An ML platform team enables careful initialization and normalization in deep models to avoid the vanishing gradient problem in training.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License