Glossary term
Glossary term
Architecture
A language model that predicts the probability of candidate tokens to fill in blanks in a sequence. For example, a masked language model can calculate probabilities for candidate word(s) to replace the underline in the following sentence:
The ____ in the hat came back.
The literature typically uses the string "MASK" instead of an underline. For example:
The "MASK" in the hat came back.
Most modern masked language models are bidirectional.
Created for this library
A legal-tech vendor pretrains a masked language model on a contract corpus to bootstrap representations for downstream clause classifiers.
A medical NLP team pretrains a masked language model on clinical notes to capture domain-specific terminology before fine-tuning.
A research team uses a masked language model as the encoder backbone of its sentence embedding pipeline.
Definition source: Google for Developers Machine Learning Glossary | Creative Commons Attribution 4.0 License