Glossary term
Glossary term
Foundations
Component that converts raw text into a sequence of integer IDs that a language model can process.
OpenAI's tiktoken library is used by the OpenAI API and LangChain to count tokens before API calls, enabling developers to stay within context limits and accurately predict costs before sending large documents.
HuggingFace Tokenizers (Rust-based) tokenises 10+ million tokens per second on CPU - used in data-preprocessing pipelines to tokenise multi-terabyte training corpora for LLM pre-training at scale.
Mistral's tokeniser uses a 32,000-token SentencePiece vocabulary optimised for European languages and code, achieving 20% fewer tokens per non-English sentence than GPT-4's tokeniser - reducing inference latency and cost for multilingual applications.