Measuring political bias in Claude

Source: Anthropic Date: 2025-11-13 URL: https://www.anthropic.com/news/political-even-handedness

Summary

Anthropic published an automated evaluation methodology for political bias in AI models, testing “political even-handedness” via 1,350 prompt pairs across 150 political topics. Results: Claude Sonnet 4.5 scored 94% on even-handedness. Benchmarked against GPT-5, Gemini 2.5 Pro, Grok 4, and Llama 4. Methodology open-sourced. Framed as establishing an industry standard for measuring AI political bias.

Implications

• Safety/policy posture thread. Political bias evaluation is directly responsive to sustained conservative criticism that AI models (especially Claude) lean left. Publishing a 94% even-handedness score with an open-source methodology is Anthropic’s empirical defense against that criticism.
• Competitive benchmarking. Testing Claude against GPT-5, Gemini 2.5 Pro, Grok 4, and Llama 4 in a Anthropic-published evaluation is a bold move — Anthropic is asserting it performs best on a benchmark it designed. The methodology being open-sourced partially addresses the obvious conflict of interest.
• 1,350 paired prompts. The scale of the evaluation (1,350 pairs, 150 topics) is meaningful — this is more rigorous than most academic political bias studies. Open-sourcing allows the AI safety community to verify the claims and extend the evaluation.
• Timing (November 2025). Post-2024 election, pre-2026 election cycle — Anthropic is getting ahead of political bias criticism before the next major election controversy.
• Watch: whether external researchers find Anthropic’s methodology advantages Claude; how Grok 4’s scores compare given Grok’s explicit “anti-woke” design; whether the 1,350 prompt pairs become an industry-standard evaluation set.