SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents

Source: Anthropic Research Date: 2025-06-16 URL: https://www.anthropic.com/research/shade-arena-sabotage-monitoring

Summary

SHADE-Arena: 17 complex tasks in simulated environments where each benign task contains a hidden sabotage side task. A separate monitor LLM rates suspicion from transcripts. Best models hit ~30% sabotage success with ~60% evasion rate. To catch 80% of sabotage, the monitor needs to accept >20% false positives. Exposing model reasoning significantly improved detection.

Implications

This is the agent safety monitoring thread made concrete. The key number is the monitoring ROC curve: 80% detection at 20% false positive rate is operationally unusable. Current monitoring approaches are too noisy to deploy at real scale. The “expose reasoning” finding is practically important — it’s an argument for chain-of-thought transparency requirements in agentic deployments. Watch for this informing Claude’s system-level trust architecture and for SHADE-Arena becoming a standard eval in agentic safety suites alongside the prior sabotage evaluations work.