Glossary term
Glossary term
Security
Manipulation of training or fine-tuning data to influence model behavior, degrade performance, create backdoors, or introduce harmful outputs. Controls include source validation, dataset change control, anomaly detection, separation of duties, provenance tracking, and review of contributed or scraped data.
Carlini et al. (USENIX 2024) demonstrated that adversaries can poison web-scale datasets like LAION and Common Crawl at low cost.
Microsoft's Tay chatbot (2016) was poisoned by Twitter users within 24 hours into producing racist outputs, an early demonstration of online learning poisoning.
Nightshade and Glaze tools from the University of Chicago let artists poison training data to protect against unauthorized scraping by image models.