CNCF: why classical observability does not work for agentic and LLM systems

What is observability and why do AI systems break the old rules?

Observability is the ability to understand the internal state of a system from external signals — logs, metrics, and traces. In classical software systems the same input always produces the same output, so anomalies are easy to spot through elevated latency or error rates. CNCF highlights that this principle simply does not hold for LLM models and AI agents operating in probabilistic environments.

Why probabilism changes everything

The same prompt can produce entirely different responses depending on temperature, context, and model state. Errors are not always technical — an agent can respond without throwing an exception but still make the wrong decision. Classical telemetry cannot see this. While Prometheus and Grafana record CPU, memory, and HTTP statuses, they miss the semantic layer: did the agent understand the task, did it follow the right step, was the result useful?

A framework for sustainable agentic observability

CNCF proposes shifting focus from infrastructure metrics to outcome reliability. Instead of ‘is the service available?’, the question becomes ‘was the decision correct?’. Concretely, this means tracking semantic prompt/response patterns, decision quality, and consistency of agent behavior over time. The approach offers a framework for ‘sustainable’ observability that does not generate data noise but instead measures what truly affects results.

Unlike deterministic microservices where a single tool covers everything, agentic systems require a new monitoring layer — one specific to LLM interactions and autonomous decisions.