Glossary term
Glossary term
Training and Fine-Tuning
A commonly used mechanism to mitigate the exploding gradient problem by artificially limiting (clipping) the maximum value of gradients when using gradient descent to train a model.
Created for this library
An ML platform team enables gradient clipping at norm 1.0 by default to prevent exploding gradients on long sequence models.
A speech recognition team adds gradient clipping during training of its recurrent acoustic model to keep updates numerically stable.
A research team enables gradient clipping when training a transformer on noisy web data to prevent occasional bad batches from destabilizing the run.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License