Examples

1. 1.

   For example, suppose the entire training set (the full batch) consists of 1,000 examples. Further suppose that you set the batch size of each mini-batch to 20. Therefore, each iteration determines the loss on a random 20 of the 1,000 examples and then adjusts the weights and biases accordingly.

2. 2.

   It is much more efficient to calculate the loss on a mini-batch than the loss on all the examples in the full batch.

3. 3.

   See Linear regression: Hyperparameters in Machine Learning Crash Course for more information.

Real-world uses

Created for this library

1. 1.

   An ML team trains with mini-batches of 256 examples to balance gradient noise and GPU utilization on a single accelerator.

2. 2.

   A computer vision team uses mini-batches of 32 images because larger batches exceed GPU memory on its target hardware.

3. 3.

   An NLP team uses mini-batches with dynamic padding to keep memory use stable across batches of varying sentence lengths.