Benchling runs multiple language models simultaneously: model disagreement surfaces data errors

Every week, a designated engineer or product manager at Benchling takes on the role of “fire chief” — reviewing AI system traces from production, looking for anomalies, and reporting to the team. This is not ceremonial: according to Nicholas Larus-Stone, Head of AI at Benchling, it is one of the key mechanisms keeping AI systems within the bounds of reliability needed for serious scientific research.

Larus-Stone appeared on LangChain’s “Max Agency” podcast with Harrison Chase, co-founder of LangChain, where he shared insights from building AI agents for a platform that stores and organizes data for biotech and pharmaceutical companies.

Why is no single model enough?

The central thesis of Larus-Stone’s approach is provocatively simple: no single frontier language model is reliable enough for tasks in scientific research where a mistake can cost weeks of work or compromise data integrity.

The solution Benchling applies in production is an ensemble — running multiple language models from different providers simultaneously on the same task. The logic is elegant:

When models agree, that is a signal of high quality. When they diverge, that is a signal that something is wrong — either with the data, the task formulation, or the underlying assumptions.

Disagreement becomes an error detector. A model does not need to know it is wrong — but another model that is wrong in a different place automatically creates a visible discrepancy that the team can investigate.

Fourteen years of structured data as a strategic advantage

Benchling is not a typical AI startup: the company has existed for more than a decade and in that time has built a platform with 14 years of structured scientific data coming directly from its clients’ laboratories.

Larus-Stone emphasizes this as a contextual advantage that generic AI cannot replicate. General language models are trained on vast amounts of publicly available text — but they are not exposed to the specific, proprietary, structured data architecture of a particular biotech company.

This contextual advantage is not just a marketing claim — it is also technical. When an AI agent has access to a 14-year history of experiments, protocols, and results in structured form, it is capable of reasoning that would otherwise require a PhD and years of laboratory experience.

Measuring quality in production

AI system evaluation is a chronic industry problem: benchmark results rarely reflect behavior in real conditions. Benchling has developed a multi-layered approach:

Weekly fire chiefs review production traces. The role rotates among engineers and product managers, meaning no single perspective is privileged.

Structured tech-ops meetings regularly analyze error patterns and trends in user feedback.

Thumbs-up/down feedback in the user interface provides a quality signal for individual responses, but Larus-Stone emphasizes this is not the primary metric source — direct trace inspection matters more.

What Benchling explicitly does not rely on: benchmark scores. Ratings on standard benchmark tests are a secondary indicator that poorly predicts behavior in a specific, highly specialized domain application.

Verifiable and non-verifiable tasks: different evaluation frameworks

Larus-Stone introduces a distinction that is rarely explicitly articulated in public discussions of AI evaluation: the difference between verifiable and non-verifiable tasks.

Code generation is verifiable: unit tests either pass or they do not. The outcome is binary and objective.

Designing an experiment — for example, which set of variables to test next week — is not verifiable in the same way. The outcome is probabilistic: it can only be judged in hindsight whether the strategy was good, and even then with considerable uncertainty.

This distinction demands completely different evaluation frameworks. Using the same metrics for both categories — as many teams do — means blindness to a significant portion of real performance.

Compressing scientific work with agents

The claim that attracts attention: according to Larus-Stone, one saved day of work with AI agents “can often become a saved week.” The compression comes from eliminating dead time between steps in an experimental workflow — waiting for results, manually moving data between tools, repetitive administrative preparation.

Even more interesting is the claim about a methodological shift: agents enable more rigorous experimental design upfront, which reduces the number of iterations required to reach a conclusion. Rather than experimenting in the dark, it becomes possible to more precisely target hypotheses.

Benchling also uses Design of Experiments (DOE) methodology combined with AI planning, suggesting the integration of statistical experimental design with language model capabilities — an approach that could be applicable well beyond biotechnology.