Glossary term
Glossary term
Foundations
A technique for handling outliers by doing either or both of the following:
Reducing feature values that are greater than a maximum threshold down to that maximum threshold.
Increasing feature values that are less than a minimum threshold up to that minimum threshold.
For example, suppose that <0.5% of values for a particular feature fall outside the range 40–60. In this case, you could do the following:
Clip all values over 60 (the maximum threshold) to be exactly 60.
Clip all values under 40 (the minimum threshold) to be exactly 40.
Outliers can damage models, sometimes causing weights to overflow during training. Some outliers can also dramatically spoil metrics like accuracy. Clipping is a common technique to limit the damage.
Gradient clipping forces gradient values within a designated range during training.
See Numerical data: Normalization in Machine Learning Crash Course for more information.
For example, suppose that <0.5% of values for a particular feature fall outside the range 40–60. In this case, you could do the following:
Clip all values over 60 (the maximum threshold) to be exactly 60.
Clip all values under 40 (the minimum threshold) to be exactly 40.
Created for this library
An ML platform team enables gradient clipping at norm 1.0 across all production training jobs to prevent exploding gradients on long sequence models.
A speech recognition vendor adds gradient clipping to stabilize training of its recurrent acoustic model on noisy field recordings.
A research team clips activations at a configurable threshold to keep mixed-precision training numerically stable.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License