LangChain: Improving Agents Is a Data Mining Problem

Vivek Trivedy of LangChain published an essay on July 7, 2026 that reframes what it actually means to “improve an AI agent.” The central thesis: improving agents is not primarily an engineering problem — it is a problem of data mining at scale.

What Are Traces and Why Are They the Currency of Improvement?

An execution trace is a structured record of every step an agent takes when solving a task: which tools were called, which results were returned, what decisions were made, and in what order. Trivedy argues that traces are “the currency of long-term agent improvement” because they translate agentic experience into a format that can be mined.

The key distinction from classical ML: traditional models produce relatively compact data records. Modern agents running multi-step tasks generate millions of multi-million-token traces. This is a fundamentally different scale of data that requires specialized processing architecture — simply reviewing traces manually or in small samples does not yield statistically meaningful improvement signals.

The Recommended Funnel: Harness, Fine-Tuning, Harness

Trivedy proposes a concrete sequence of work he describes as a “sandwich” approach:

Harness engineering → Fine-tuning → Harness engineering

The harness is everything that surrounds the model: prompts, tools, memory, retry logic, the orchestration layer. Trivedy argues that harness engineering alone often improves performance sufficiently for most teams — without any modification of model weights. The harness provides a “high-throughput surface for transferring knowledge” into the agent without the cost of fine-tuning.

Fine-tuning comes only in the second step, and only when harness optimization has plateaued. After fine-tuning, the third step is returning to harness optimization with the improved model as the base.

Counterintuitive Advice: Deploy Agents Early

One of the essay’s central pieces of advice runs directly counter to the intuition of most development teams: deploy agents early, before they are “finished”, even into limited production.

The reasoning is pragmatic. Without real user interactions, there are no traces. Without traces, there is no material to mine. Without mining, there are no signals about where and why the agent makes mistakes. Closed-loop optimization — iterating over synthetic data or small test sets — is blind compared to mining millions of real interactions.

Early deployment is not a call for irresponsibility — it is a strategic decision to start the data collection loop as early as possible, because that loop determines the rate of improvement in all subsequent phases.

The Dual Problem: Cost and Context

Trivedy frames modern agent data as a dual problem. First is cost — the volume of tokens in traces is enormous, and processing every trace with a frontier model quickly becomes economically unsustainable. Second is context — uncovering useful improvement signals within vast collections of traces is not a trivial task.

The solution to the cost problem: fine-tuned smaller open models. For narrow tasks such as trace analysis and error pattern detection, fine-tuned smaller models outperform frontier models at dramatically lower cost per token. This is especially relevant because trace analysis is itself a narrow and well-defined task — ideal for fine-tuning.

Evals as Training Data

Trivedy proposes a reframing of evaluations: evals are not merely a success metric, they are training data for agents. Passing through an eval transfers measured behaviors into agent performance. Evaluation design becomes critical infrastructure, not an afterthought.

This perspective shifts team priorities — investing in high-quality eval design from the early stages of a project is not an administrative burden but a direct investment in the capacity for continuous agent improvement.

Compound Agent Systems for Autonomous Improvement

The essay describes an advanced pattern: compound agent systems that autonomously read traces, identify problems, generate fixes, create evals, and write insights into memory stores. An agent improves an agent — a loop that accelerates as more traces become available.

This is not a speculative vision but a logical extension of the approach Trivedy describes — each step in the funnel can be automated by an agent that receives traces as input and produces improvements as output.

Conclusion: Observability and Continuous Learning Are One

Trivedy concludes that observability and continuous learning are two sides of the same coin. Every organization that systematically improves agents must be an observability organization — and vice versa. Teams building agents without infrastructure for collecting and mining traces are operating without feedback and are impoverishing their ability to improve.