Glossary term
Glossary term
Foundations
Language-agnostic subword tokenisation library supporting BPE and unigram language model tokenisation, used for multilingual models.
Google uses SentencePiece to tokenise training data for T5, mT5, PaLM, and Gemini, enabling consistent multilingual tokenisation across 100+ languages without language-specific pre-processing.
Meta used SentencePiece BPE to train Llama 2's tokeniser on a byte-level vocabulary, ensuring that all Unicode characters (including rare scripts) can be represented without unknown token issues.
NLLB (Meta, No Language Left Behind) uses a SentencePiece model trained jointly on 200 languages, producing a shared vocabulary that balances tokenisation efficiency across all languages without favouring high-resource languages.