Glossary term
Glossary term
Training and Fine-Tuning
A backpropagation technique that updates the parameters only once per epoch rather than once per iteration. After processing each mini-batch, gradient accumulation simply updates a running total of gradients. Then, after processing the last mini-batch in the epoch, the system finally updates the parameters based on the total of all gradient changes.
Gradient accumulation is useful when the batch size is very large compared to the amount of available memory for training. When memory is an issue, the natural tendency is to reduce batch size. However, reducing the batch size in normal backpropagation increases the number of parameter updates. Gradient accumulation enables the model to avoid memory issues but still train efficiently.
Created for this library
A vision team uses gradient accumulation to mimic a large effective batch size when its GPUs cannot fit the full batch in memory.
A speech recognition team uses gradient accumulation over four steps to reach an effective batch size that produced better convergence in earlier experiments.
An NLP team uses gradient accumulation to simulate large-batch training on a small cluster without renting larger machines.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License