Is it agentic enough? Benchmarking open models on your own tooling

Source: HuggingFace Date: 2026-06-18 URL: https://huggingface.co/blog/is-it-agentic-enough

Summary

HuggingFace published agent-eval, an evaluation harness that measures not just whether agents complete tasks but the effort cost — tokens, turns, and time. The framework tests models across three tooling tiers (bare, full source clone, curated skill documentation) using the transformers library as a concrete case study. The headline finding: a CLI addition that reduced completion time for 70B+ models actively broke smaller models (e.g. Qwen3-14B), which misread CLI documentation as a callable tool rather than shell commands — a regression that final-answer-only benchmarks would completely miss.

Implications

• Agentic engineering patterns — eval methodology. The model-size-dependent regression is a significant finding: tooling improvements are not monotonically beneficial across capability tiers. Teams deploying smaller models in agentic loops need their own evals; assuming that results from large-model benchmarks transfer downward is demonstrably unsafe. The effort-cost framing (not just pass/fail) is also directionally correct for production systems where token cost and latency matter.
• Coding-agent competition. The transformers case study is directly relevant to coding agent design: agent-facing APIs (CLIs, SDKs, documentation structure) need to be tested across the model size range that will consume them, not just on the largest available model. This is an argument for API providers to publish agentic benchmarks alongside their SDKs.
• Model landscape — open model capability calibration. The gap between 14B and 70B models on agentic tasks is larger than aggregate benchmarks suggest. Teams choosing between smaller (cheaper, faster, local) and larger models for agent deployment now have a concrete methodology for making that decision on their own tooling, not on generic benchmarks.