Glossary term
Glossary term
Foundations
A function that enables neural networks to learn nonlinear (complex) relationships between features and the label.
Popular activation functions include:
The plots of activation functions are never single straight lines. For example, the plot of the ReLU activation function consists of two straight lines:
A plot of the sigmoid activation function looks as follows:
Click the icon to see an example.
See Neural networks: Activation functions in Machine Learning Crash Course for more information.
Created for this library
An e-commerce ranking team swaps sigmoid for ReLU in the hidden layers of its click model and observes faster convergence with a small lift in offline AUC.
A medical imaging startup adopts GELU in its diagnostic CNN after benchmarking activation functions on its in-house dataset of chest X-rays.
A speech recognition vendor relies on Swish activations in its acoustic model after finding it reduces word error rate on low-resource languages.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License