Observability

With agents, clean logs mean nothing. An agent can be successfully failing: polished, confident, wrong, every metric green. "Did it work?" is no longer a system question but a question of meaning.

The unit of observation is the full session, not the request. A failure in step three surfaces nowhere: the telemetry stays green while the wrong output travels downstream. Semantic checks ask what the logs cannot.

Why agents break monitoring

In the traditional software paradigm, if the logs are clean, the system works. With agents, clean logs mean nothing. Determinism is the first deviation: the same prompt can return different outputs, so we cannot test by replaying inputs. Silent failures are a real possibility: the agent confidently reports success while the content is wrong. In multi-step causal chains, a failure in step n might not surface until many steps later, so no single step can be diagnosed alone; full-session trace capture across the entire reasoning chain is required. And most importantly, there is semantic quality: we need to ascertain whether the retrieved context is relevant, whether the agent understood the query, whether the output was accurate and policy-compliant. Traditional monitoring has no concept for this.

The failure classes

Without proper tracing in place, several classes of failure can occur: recursive loops that generate hundreds of API calls for a single task while appearing successful; hallucinated tool arguments matching training patterns rather than actual API schemas; instruction drift, where over long sessions the system prompt loses weight as recent tokens dominate attention; context overload, where the right documents were fetched and the agent never used them; and masked backend crashes, where the agent misinterprets errors and reports tasks as completed when the backend failed.

What to monitor

Taken together, agent observability must track whether the agent understood the query, whether the retrieved context was relevant, whether the right tool calls were made and succeeded, whether the output was accurate and aligned with policy, and the full session, grouping related interactions to reveal coherence issues across turns. NIST AI 800-4 gives the six-category frame shown at the anatomy level: functionality, operational, human factors, security, compliance and large-scale impacts.