Threat Landscape & Frameworks
Overview
AI threats don't fit neatly into traditional cybersecurity taxonomies. They span the entire ML pipeline — from training data to inference output — and require frameworks designed specifically for machine learning systems.
Threat Actor Profiles
| Actor | Motivation | Typical Attacks | Resources |
|---|---|---|---|
| Script kiddie | Curiosity, bragging rights | Known jailbreaks, copy-paste injection | Low — public tools only |
| Red teamer | Authorized testing | Full methodology, custom tooling | Medium-High — scoped access |
| Cybercriminal | Financial gain | AI-powered phishing, deepfakes, fraud | Medium — cloud compute, social engineering |
| Competitor | IP theft | Model extraction, training data theft | High — funded research teams |
| Nation-state | Espionage, disruption | Data poisoning, supply chain, influence ops | Very High — custom labs, insider access |
| Insider | Varies | Training data manipulation, model backdoors | High — direct pipeline access |
Key Frameworks
Two frameworks matter most for AI red teaming:
OWASP LLM Top 10
Focuses on application-level vulnerabilities in LLM deployments. Best for scoping pentests and communicating risk to developers.
MITRE ATLAS
Focuses on adversarial tactics and techniques across the ML lifecycle. ATT&CK-style matrix for machine learning. Best for threat modeling and mapping attack paths.
Mapping to the Kill Chain
| Cyber Kill Chain Phase | AI-Specific Activity |
|---|---|
| Reconnaissance | Fingerprint model, extract system prompt, enumerate tools |
| Weaponization | Craft adversarial prompts, build injection payloads, fine-tune attack model |
| Delivery | Plant indirect injection in documents, web pages, emails |
| Exploitation | Execute prompt injection, jailbreak, trigger backdoor |
| Installation | Achieve persistence via poisoned RAG source, tool manipulation |
| Command & Control | Exfiltrate data via tool calls, establish ongoing injection channel |
| Actions on Objectives | Data theft, unauthorized actions, model compromise, disinformation |