LangChain: The agent that fixes agents — how LangSmith Engine was built

LangChain published a detailed technical overview of LangSmith Engine — an autonomous agent that analyzes errors in production AI agents and proposes concrete fixes.

What is LangSmith and what is LangSmith Engine?

LangSmith is an AI agent engineering platform offering observability, evaluation, and fleet management for production agents. LangSmith Engine is a meta-agent built on top of that platform: it continuously reviews trace data from deployed agents, detects recurring error patterns, and automatically proposes evaluators and regression examples.

In short: it is an agent whose only job is to improve other agents.

How does the “agent-improving-agent” architecture work?

The engine operates through a multi-phase pipeline. First, it compresses thousands of traces into compact trajectories (role, tool name, latency, character count) to avoid context overflow. Then a screener sub-agent quickly classifies each trace as clean or suspicious, while investigator sub-agents do deep analysis only on flagged cases.

Errors are constrained to a predefined list of categories — agent_looping, incorrect_tool_args, missing_tool, pii_leak — which keeps quality under control. For each problem found, the Engine generates an evaluator (code-based or LLM-as-judge), validates it with the test_evaluator tool on real traces, and submits it to the Issue Board with a severity level.

Why this matters for development teams

Until now, debugging AI agents required manual log review and subjective assessment. The Engine automates that process end-to-end — from detection to proposed regression tests with assertions. Teams managing agent fleets can identify systemic issues without manually reviewing hundreds of traces. The approach is a good example of how meta-agents are becoming a standard part of MLOps infrastructure.