arXiv:2606.14350 — 8 Patterns for Compound AI Systems: Up to 71% Lower Costs

Monolithic Models Lose on Flexibility

A single large language model chosen at design time cannot easily adapt its behavior to the complexity of the input task, nor distribute work among specialized components. That is the starting point for researchers Miloš Gravara, Andrija Stanisić, and Stefan Nastić from TU Wien, who propose a systematic methodology for designing compound AI systems that orchestrate multiple models, algorithms, and tools through explicit control logic.

The argument is not merely theoretical. As the complexity of production AI systems grows, ad hoc addition of components without a formal framework leads to suboptimal configurations and unpredictable trade-offs between latency, accuracy, and cost.

What Is the Real Performance Difference Between Compound and Monolithic Systems?

Results from three empirical case studies provide a concrete answer. Compound configurations achieved monolithic model accuracy within a range of 2.5 to 4 percentage points — while simultaneously reducing latency by up to 60% and cutting costs by up to 71%. These are measurable trade-offs that are consistent across different task types, making the compound approach economically attractive for nearly all production use cases where absolute maximum accuracy is not the sole selection criterion.

Two Degrees of Freedom, Eight Patterns

The methodology organizes the design space along two degrees of freedom: workflow topology and configuration selection. By combining these dimensions, the authors identify 8 design patterns that address specific shortcomings of the single-model approach.

Patterns include chained configurations where sequential flow passes through specialized components, each optimized for a subtask. Routers dynamically redirect tasks based on estimated complexity or input type. Ensembles and critics combine parallel execution with subsequent scoring or voting among outputs from multiple models. Each pattern carries explicitly documented trade-offs that enable systematic architecture selection instead of intuitive guessing.

The Key Tension: Model and Configuration Selection Together

The authors emphasize that model selection and configuration parameters together determine system performance. This insight sounds trivial but has serious engineering consequences: as the number of components increases, the space of possible configurations grows combinatorially, making manual tuning impractical in production environments with demanding SLO compliance requirements.

That combinatorial explosion motivates the formalized methodology. Without a structured search space, engineers rely on experience and ad hoc experimentation, leading to configurations that may work but are far from optimal for a given set of constraints.

Five Open Challenges

The paper concludes by identifying five open challenges that point toward future research. The hardest is automated discovery of configurations that comply with Service Level Objectives — mechanisms that would autonomously find combinations of components, parameters, and topology satisfying latency, cost, and accuracy requirements without manual intervention.

Other challenges include managing runtime adaptation (dynamically changing configuration during execution in response to varying input difficulty), standardized evaluation of different compound architectures, and formal SLO compliance guarantees in distributed environments. These open problems trace the boundary between today’s heuristic solutions and systematic AI infrastructure engineering.

Why This Matters Now

The “compound AI” paradigm — chains, routers, ensembles, critics — is rapidly becoming the dominant architecture in production AI systems. Tools such as LangChain, DSPy, or LlamaIndex already implement many of these patterns, but without a formal framework for design and evaluation that would ensure predictable trade-offs.

The work by Gravara and colleagues takes a step toward disciplined engineering of compound AI systems — with measurable, reproducible trade-offs instead of guesswork. For production teams today choosing between a monolithic and a compound approach, these 8 patterns and three concrete measurements provide a rare empirical anchor for such decisions.