16 million stolen conversations, 24,000 fake accounts, and what it means for the AI tools you’re choosing this week + the manifold probe

Source: Nate’s Newsletter Date: 2026-02-25 URL: https://natesnewsletter.substack.com/p/three-labs-just-stole-claudes-brain

Summary

Three Chinese AI labs extracted Claude’s capabilities at scale by creating 24,000 fake API accounts and harvesting 16 million conversations for model distillation. Nate frames this as an economic force rather than geopolitics: distillation is structural piracy driven by the same incentives affecting any lab, with Meta’s talent and data strategies as the Western parallel. Distilled models score well on benchmarks but degrade on sustained agentic work — which is where real business value concentrates. The “Manifold Probe” diagnostic tests models against domain-specific tasks rather than leaderboard scores to reveal those gaps.

Implications

• Validates the case for evaluating models against real workflow tasks rather than published benchmarks — synthetic scores are gameable, agentic robustness is not.
• Benchmark collapse on sustained autonomous work makes distillation less threatening to frontier labs in the short term, but accelerates the race to produce “agentic reliability” as the next defensible moat.
• API-level capability extraction is now a standard competitive vector; rate limiting and account verification become meaningful infrastructure decisions, not just cost controls.
• Feeds the thread on trust signals in tool selection: cheapest-available-model reasoning breaks down precisely where agent workloads are highest-stakes.