Glossary term
Glossary term
Training and Fine-Tuning
An undesirable behavior in which neurons predict patterns in training data by relying almost exclusively on outputs of specific other neurons instead of relying on the network's behavior as a whole. When the patterns that cause co-adaptation are not present in validation data, then co-adaptation causes overfitting. Dropout regularization reduces co-adaptation because dropout ensures neurons cannot rely solely on specific other neurons.
Created for this library
A computer vision team adds dropout to break co-adaptation between neurons so the model does not rely on a small clique of features.
A speech recognition team monitors hidden activations for co-adaptation patterns and increases dropout when groups of neurons fire together too consistently.
An NLP team uses high dropout in the embedding layer of a small model to reduce co-adaptation when training data is limited.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License