Glossary term
Glossary term
Foundations
The variables that you or a hyperparameter tuning service adjust during successive runs of training a model. For example, learning rate is a hyperparameter. You could set the learning rate to 0.01 before one training session. If you determine that 0.01 is too high, you could perhaps set the learning rate to 0.003 for the next training session.
In contrast, parameters are the various weights and bias that the model learns during training.
See Linear regression: Hyperparameters in Machine Learning Crash Course for more information.
Created for this library
An ML team tunes learning rate as the most impactful hyperparameter in its training pipeline before tuning batch size or dropout.
A research team runs a hyperparameter sweep over learning rate and weight decay to find a stable configuration for a new architecture.
An ML platform team standardizes hyperparameter defaults per model family so engineers can focus on data quality and features.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License