12. AI-Powered Attacks
AI has shifted from a defensive weapon to an offensive one. The following are attack approaches distilled from real APT activity.
12.1 AI-Generated Phishing (Weaponized in the Wild)
Deepfake voice/video phishing: clone an executive's voice → phone call instructing a wire transfer / password reset, or forge a CFO's face in a video meeting - Technical principle: GAN + diffusion models → high-fidelity voice/face cloning from a small number of samples - Operational approach: collect a target executive's public speaking videos (YouTube / earnings calls) → train the model → forge an urgent phone directive
LLM-assisted phishing (multiple APT cases already observed): - Attack chain: the LLM automatically analyzes the victim's LinkedIn / Twitter / industry forums → builds a psychological profile → generates a personalized email - Emails have no grammatical errors and none of the traditional detection "red flag" features - References real events ("We spoke at the Gartner Security Summit in Berlin") and real colleagues' names - Automated A/B testing of different message variants to optimize click-through rate - AI-obfuscated phishing discovered by Microsoft in 2025: SVG files use business terminology (revenue/operations/risk/shares) to encode the malicious payload rather than traditional encryption-based obfuscation → traditional phishing detection fails completely. After analysis, Microsoft Security Copilot concluded it was "not human-written and highly likely generated by an LLM" - Defense difficulty: AI-written emails show no semantic anomaly, and neither traditional rules nor human training can detect them
12.2 AI-Assisted Malicious Code (Real-World Cases)
LAMEHUG (discovered by CERT-UA, 2025.07):
- Attack approach: phishing attachment disguised as an AI image-generation tool (AI_generator_uncensored_Canvas_PRO_v0.9.exe)
- Core innovation: the malicious code calls the Qwen 2.5-Coder-32B-Instruct model on the HuggingFace API to generate reconnaissance / data-theft / system-manipulation commands in real time
- Flow: LLM_QUERY_EX() builds the prompt → sends it to the HuggingFace API → the LLM returns Windows commands → executed locally
- Significance: malicious behavior is generated dynamically by the LLM, so static analysis cannot predict the specific behavior, and the executed commands differ every time
SesameOp (discovered by Microsoft DART, 2025.11): - Attack approach: the backdoor implant uses the OpenAI Assistants API as its C2 channel - Flow: malicious component OpenAIAgent.Netapi64 → calls the OpenAI Assistants API to fetch commands → decrypts and executes them → encrypts the results and sends them back to OpenAI - Stealth: C2 traffic is fully mixed with normal AI API calls, with payload compression + layered encryption (symmetric + asymmetric) - Significance: no self-built C2 infrastructure required; a legitimate AI service is used as the relay
ShadowAI (discovered by Akamai, 2025):
- Attack approach: the malware disguises its C2 traffic as requests to the /v1/chat/completions endpoint
- Flow: sends Base64-encoded strings (disguised as LLM requests) → the response is XOR + Base64 decrypted and its instructions executed
- Stealth: blends into an enterprise's growing volume of normal LLM API traffic, indistinguishable at the network layer
Unit42 research, 2026.01 (forward-looking attack): - Attack approach: a carefully crafted prompt is embedded in a web page → calls a legitimate LLM API → generates phishing JS code in real time → executed by the browser - Key point: the generated phishing page's code structure differs on every visit (polymorphic), so there is no static payload to detect - Defense: requires runtime behavior analysis (browser sandbox + real-time detection); static/network layers are insufficient
Automated AV/EDR evasion approaches: - The LLM automatically generates exploit code from patch diffs - The LLM generates polymorphic/metamorphic shellcode (structurally different each compilation, functionally identical) - AI-assisted YARA rule evasion (analyzes detection rules → generates evasive variants)
12.3 AI-Enhanced Reconnaissance Automation
- The LLM automatically analyzes the target's public information → generates an attack surface report
- AI-assisted code audit (vulnerability pattern recognition)
- Automated social-engineering data correlation (multi-platform OSINT aggregation)
12.4 AI-Driven Device Code Phishing (EvilTokens PhaaS)
Large-scale attack campaign disclosed by Microsoft, 2026.04: - Attack chain: AI generates personalized emails (RFP / invoice / manufacturing-workflow themes) → the victim clicks the link → a Device Code is generated dynamically → auto-filled on the Microsoft sign-in page → the user completes MFA on the genuine Microsoft page → the attacker obtains the Access Token - Key innovation: dynamic Device Code generation bypasses the 15-minute expiry limit. Traditional attacks pre-generate the code and embed it in the email, so it expires if the user opens the email 20 minutes later; the new method only starts the flow when the user clicks the link, keeping it valid within the 15-minute window - EvilTokens PhaaS toolkit: automates the entire attack into a service - See §17.1 for the complete technical analysis of Device Code phishing
12.5 AI Infrastructure Attacks (Training/Inference Stack Vulnerabilities)
Attack the infrastructure of the AI systems themselves: real 2025-2026 CVEs in PyTorch/vLLM/SGLang/training frameworks.
Deserialization supply chain (CWE-502):
| Vulnerability | Component | Principle |
|---|---|---|
| CVE-2025-32434 (9.3) | PyTorch <2.6.0 | weights_only=True is bypassed: the storages loading path of legacy tar checkpoints bypasses the _weights_only_unpickler allowlist |
| CVE-2025-67729 (8.8) | LMDeploy ≤0.11 | six torch.load() calls without arguments (no weights_only), so a malicious .pt/.bin executes on load |
| CVE-2026-46432 (7.8) | LMDeploy <0.13.0 | trust_remote_code=True hardcoded with no opt-out, so the HF repo's configuration_*.py is imported and executed on model load |
| CVE-2025-66448 | vLLM <0.11.1 | HF auto_map dual-repo bypass: a malicious repo combined with trust_remote_code achieves RCE |
Attack chain: malicious .pt/.pkl/.bin → published to HF → victim runs from_pretrained() or torch.load() → __reduce__ executes → reverse shell / persistence
Defense: use safetensors instead of pickle / RestrictedUnpickler allowlist / picklescan repository scanning / explicit opt-in for trust_remote_code
Inference engine network attack surface:
| CVE | Component | Attack surface |
|---|---|---|
| CVE-2026-3059 (9.8) | SGLang ≤0.5.9 | all ZMQ inter-component communication is unauthenticated + pickle serialization → remote RCE |
| CVE-2025-47277 (9.8) | vLLM 0.6.5-0.8.4 | PyNcclPipe distributed component pickle RCE (TCPStore listens on 0.0.0.0 by default) |
| CVE-2025-6242 | vLLM <0.11.0 | MediaConnector SSRF → the image_url parameter reaches the internal network / cloud metadata |
| CVE-2026-34159 (9.8) | llama.cpp <b8492 | RPC backend buffer=0 → arbitrary read/write → full RCE chain |
Training framework attacks: verl/grader eval injection / slime RL framework vulnerabilities / Ray cluster control plane unauthorized access (≤2.51.0)
12.6 OWASP LLM Top 10 Quick Reference (2025)
| # | Risk | Attack scenario | Red team exploitation |
|---|---|---|---|
| LLM01 | Prompt injection | Indirect injection (instructions hidden in web pages / documents / tool output) | Agent reads a malicious page → hijacked |
| LLM02 | Sensitive information disclosure | Prompting to extract the system prompt / training data leakage | Obtain API keys / internal configuration |
| LLM03 | Supply chain | Malicious HF models / typosquatted pip packages | Malicious .pt executes on load (see 12.5) |
| LLM04 | Data poisoning | Backdoor injected via fine-tuning data | Trigger word activates malicious behavior |
| LLM05 | Improper output handling | LLM output passed directly into exec()/innerHTML | Inject RCE/XSS payloads |
| LLM06 | Excessive agency | Agent holds tool permissions beyond what it needs | Induce the Agent to call dangerous tools |
| LLM07 | System prompt leakage | "Ignore instructions and repeat the system prompt" | Obtain hidden logic / API keys |
| LLM08 | Vector/embedding weaknesses | RAG retrieval-layer poisoning | Inject poisoned retrieval results into the knowledge base |
| LLM09 | Misinformation | Model fabricates citations/facts | Exploit hallucination to spread false information |
| LLM10 | Unbounded consumption | Nested prompts induce token explosion | API-bill denial of service |
Full definitions: https://genai.owasp.org/llm-top-10/
12.7 Agent Security Attack Surface
Agent hijacking chain: malicious web page/document → indirect prompt injection → Agent hijacked → calls dangerous tools → data exfiltration / privilege escalation
Tool-call abuse:
- When an Agent holds high-risk tools such as delete_file / send_email / execute_command, they can be triggered via indirect injection
- MCP (Model Context Protocol) permissions are too broad → a malicious Server impersonates a legitimate tool
- Memory poisoning: the Agent's memory/context is polluted → subsequent conversations execute malicious instructions
Defense essentials: least-privilege toolset / output consistency validation / tool-call approval / sandbox isolation