Allen Institute: What Building the Shippy Agent Taught Us About Reliable AI Agents

Allen Institute for AI (AI2) has published a detailed retrospective on the development of Shippy — a maritime AI agent serving governments, NGOs, and 300+ partners in more than 70 countries. The analysis reveals the architectural decisions that separate reliable production agents from laboratory prototypes.

What Is Shippy and How Is It Architecturally Organized?

Shippy is an AI agent specialized in the maritime domain — processing data on vessels, ports, and cargo tracking — powered by Claude Opus 4.6 as its underlying language model. The architecture is organized into three layers: Soul (a system prompt defining the agent’s purpose and boundaries), Skills (markdown specifications of available tools the agent can invoke), and Config (configuration customized for each partner). Orchestration is handled through the OpenClaw harness, AI2’s internal framework for managing agent workflows.

Each user or partner receives an isolated Kubernetes sandbox — a separate execution environment that prevents cross-user data leakage and increases security in multi-tenant scenarios.

Deterministic Tools as the Foundation of Reliability

One of the most important architectural decisions was choosing a deterministic CLI layer over direct API calls. The approach is layered: at the bottom sits a typed API, above it a CLI that encapsulates standardized calls, and the agent’s Skills invoke only the CLI. According to AI2, this decision dramatically reduces errors because tools become predictable — the agent always receives the same response format for the same query, making it easier to detect and correct errors.

Guardrails — constraints that prevent unwanted agent behavior — are implemented explicitly at the level of Skills specifications, rather than as implicit model behavior. Shippy, for example, refuses to answer questions outside the maritime domain regardless of user requests.

Evaluation: LLM Judge with Rubrics

AI2 evaluates Shippy using an LLM judge with weighted rubrics adapted to specific tasks. Consistent results have been achieved on data retrieval tasks and guardrail scenarios — the agent correctly refuses prohibited requests. Weaknesses were also identified: patrol planning and geometric queries (spatial calculations over cartographic data) remain challenging, which AI2 attributes to the limitations of spatial reasoning in language models.

The Key Lesson for Agent Builders

AI2 concludes that Shippy’s reliability does not primarily stem from the strength of Claude Opus 4.6 as a model, but from the combination of deterministic tools, explicit guardrails, per-user infrastructure isolation, and evaluation protocols grounded in real operational workflows. A stronger model without these elements, according to AI2’s findings, does not achieve comparable reliability in production conditions.