Glossary term
Glossary term
Infrastructure and Serving
In ML parallel programming, a term associated with assigning the data and model to TPU chips, and defining how these values will be sharded or replicated.
Mesh is an overloaded term that can mean either of the following:
A physical layout of TPU chips.
An abstract logical construct for mapping the data and model to the TPU chips.
In either case, a mesh is specified as a shape.
Created for this library
An ML platform team configures a 2D device mesh on its TPU pod to combine model and data parallelism for its largest training job.
A research team uses an explicit device mesh in JAX to control how parameters and activations are sharded across accelerators.
An ML platform team chooses mesh sizes that balance model and data parallelism for its largest training jobs.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License