Glossary term
Glossary term
Training and Fine-Tuning
Family of techniques that fine-tune a small subset of parameters or add lightweight modules to adapt a model without updating all weights.
HuggingFace PEFT library is used by 100,000+ researchers to fine-tune Llama, Mistral, and Falcon models using LoRA, QLoRA, and Adapters - reducing GPU memory requirements from 80GB to 16GB for 7B models.
Anthropic uses PEFT techniques to create domain-specific Claude variants for enterprise customers - a financial institution receives a customised model trained via LoRA on its internal documentation without the full 70B fine-tuning cost.
DoRA (Liu et al. 2024) decomposes pre-trained weights into magnitude and direction components for PEFT - achieving SFT-comparable quality at LoRA-comparable cost, adopted in HuggingFace PEFT library v0.7.