Glossary term
Glossary term
Security
Misuse that can reasonably be anticipated based on system capabilities, user incentives, operating context, threat models, or prior incidents. Foreseeable misuse should be documented during design and reassessed when new capabilities, integrations, user groups, or threat intelligence emerge.
A Chevrolet dealership chatbot was tricked into agreeing to sell a 2024 Tahoe for USD 1 in December 2023, a foreseeable misuse pattern documented across multiple dealer bots.
Foreseeable misuse of voice cloning models for vishing was demonstrated by IBM X-Force in 2023 and led to OpenAI delaying Voice Engine public release.
The EU AI Act Article 9 requires high-risk AI providers to identify and mitigate reasonably foreseeable misuse as part of the risk management system.