LangChain: How We Made Our Documentation Test Itself Using Deep Agents

Every developer knows the frustration: you copy a code example from the official documentation, run it, and — it doesn’t work. The API changed, a parameter was renamed, but the documentation wasn’t updated. LangChain decided to solve that problem using its own AI agents.

How Does the Documentation Self-Testing System Work?

LangChain uses Deep Agents — AI agents with multiple specialized skills organized in a .deepagents/skills/ directory. The system automatically traverses the documentation, identifies all inline code snippets, and migrates them into standalone test files.

Each extracted snippet becomes an executable test that runs through GitHub Actions on every commit. If a test fails — because the API changed — the system automatically regenerates a verified code snippet that reflects the current state of the API and updates the documentation.

Why Is This Relevant Beyond LangChain?

The “documentation as code with tests” approach is not new, but using AI agents to automate the entire cycle is. Traditionally, testing documentation requires manually writing tests for each example — work that is rarely maintained because it has no direct impact on product functionality.

LangChain’s approach treats documentation with the same rigor as production code: every example must pass a test, every test runs automatically, every failure is automatically fixed. For projects with fast development cycles — where APIs change weekly — this is the difference between documentation that works and documentation that frustrates.

The system is available as a reference implementation that other open-source projects can adapt to their own needs.