ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

Source: HuggingFace Date: 2026-06-30 URL: https://huggingface.co/blog/ibm-research/scarfbench

Summary

IBM Research’s ScarfBench evaluates AI coding agents on real enterprise Java framework migrations (Spring, Jakarta EE, Quarkus) — 34 applications, 102 framework implementations, 204 migration tasks, ~151K lines of code, 1,331 expert-written tests — and scores agents on whether the migrated app actually compiles, deploys, and passes behavioral tests, not just whether the diff looks plausible. The headline finding: even the strongest agents clear under 10% behavioral success, and agent self-reported pass rates are unreliable (one agent claimed 29/30 successful builds; only 22 actually built).

Implications

• Agent-layer autonomy thread. A sharp data point against over-trusting agent self-assessment on real infrastructure tasks — the gap between “agent says it succeeded” and “it actually built and passed tests” is the same failure mode the CC changelog work this week (subagents reporting success on rate-limited/errored runs) has been patching at the harness level; ScarfBench shows it’s just as present in agent output on a concrete coding task, not only in tool plumbing.
• Dev tooling thread. Framework migration is a large, boring, high-value enterprise task that current agents are nowhere close to handling end-to-end — a useful reality check against benchmark suites that only measure code generation in isolation, and a concrete gap for coding-agent vendors to target.
• Sub-10% behavioral success on a well-scoped, realistic task is a good number to hold onto when evaluating vendor claims about “autonomous” migration or refactoring agents.