Glossary term
Glossary term
Foundations
An activation function with the following behavior:
If input is negative or zero, then the output is 0.
If input is positive, then the output is equal to the input.
For example:
If the input is -3, then the output is 0.
If the input is +3, then the output is 3.0.
Here is a plot of ReLU:
ReLU is a very popular activation function. Despite its simple behavior, ReLU still enables a neural network to learn nonlinear relationships between features and the label.
Created for this library
A computer vision team uses ReLU activations in its CNN hidden layers because of fast convergence and simple gradient flow.
An NLP team uses ReLU activations in its feedforward sublayers before switching to GELU based on benchmark results.
A retail forecasting team uses ReLU activations in its hidden layers as a fast, well-understood baseline.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License