What building Shippy taught us about building agents

Source: HuggingFace Date: 2026-07-15 URL: https://huggingface.co/blog/allenai/shippy-tech-blog

Summary

Allen Institute for AI (Ai2) published lessons from building Shippy, a maritime domain-awareness agent that analyzes vessel behavior and fishing activity from live satellite/vessel data (via the Skylight platform) for ocean-protection partners in 70 countries. Key architectural choices: a “soul/skills/config” separation for the agent (system prompt, task capabilities, runtime settings versioned independently), a purpose-built deterministic CLI in front of live APIs (handling auth, pagination, validation, writing results to files instead of stdout to avoid buffer limits) rather than giving the agent raw API access, per-session isolated Kubernetes deployments for multi-tenant data isolation, and a custom evaluation framework (Harbor) where domain experts grade whole-agent behavior against live data instead of static benchmarks.

Implications

• Agent-runtime hardening thread: the “deterministic CLI wrapper instead of raw API access” pattern is a concrete, independently-arrived-at example of the same instinct driving harness sandboxing elsewhere (fnox, Claude Code worktree confinement) — treat the agent as a process to constrain, not a trusted user.
• Evaluation methodology: another data point (alongside VoiceEQ and the model-routing piece below) that static benchmarks are being displaced by live, domain-expert-graded evaluation for production agents — a recurring theme across this batch of signals.
• Production-scale agent deployment: 300+ partners, 70 countries, isolated per-session compute is a real-world scale reference for what “agent in production” infrastructure looks like outside the coding-agent context that dominates most of this radar.