Glossary term
Glossary term
Infrastructure and Serving
A parallelism technique where the same computation is run on different input data in parallel on different devices. The goal of SPMD is to obtain results more quickly. It is the most common style of parallel programming.
Created for this library
A research engineer uses SPMD in JAX to run the same training program across multiple devices, each operating on its shard of data.
An ML platform team uses SPMD-style sharding in its largest training jobs to scale across TPU pods.
A research team uses SPMD with pjit to express partitioning declaratively over a device mesh.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License