Observability from day one, not week ten.

The first production incident with an agent is almost always the same: a user reports something weird, the team opens the logs, and the logs say the request succeeded. Status 200. Latency 1.4s. No error. The agent simply did the wrong thing, with confidence, and the standard observability stack has no opinion on whether that's a failure or not.

This is why we treat observability as a launch blocker, not a polish item. If we can't see what the agent did and why, we can't operate it. The dashboard ships before the feature does.

What standard APM misses

Datadog, New Relic, OpenTelemetry — all great tools, all designed for a world where "the request worked" and "the response was correct" are the same question. Agents break that assumption. A correct-looking response can be wrong. An incorrect-looking response can be the right escalation. The interesting failures aren't 500s; they're 200s that should have been escalations, refusals, or different tool calls.

We instrument three layers on top of standard tracing, every time:

• The decision layer. What did the model decide to do at each step? Which tool, which arguments, which branch? Decisions are first-class spans, not log lines.
• The grounding layer. What context did the model see when it made each decision? Which retrieved chunks, which prior messages, which system fields? Without this, you can't reconstruct why a decision was made.
• The outcome layer. What happened after the decision? Did the tool call succeed? Did the user accept the output? Did a human override it? Outcomes close the loop on whether the decision was good.

The five signals we put on every agent dashboard

On day one, before launch, every agent has a dashboard with five panels. Not ten. Five. More than that, and operators stop looking.

• Coverage. What percentage of traffic is the agent actually handling end-to-end, versus escalating, versus failing? This is the headline metric.
• Escalation reason mix. Of the escalations, what categories are they in? A shift in the mix is the earliest warning that the world has changed.
• Tool error rate, per tool. Tools fail. Models react badly to tool failures. You want to see tool-level reliability before users do.
• Eval pass rate on a rolling production sample. A small slice of production traffic gets re-graded against the eval suite. If pass rate drifts down, something has changed — the data, the model, or a silent prompt edit.
• Cost per resolved interaction. Not cost per token. Cost per actual unit of work delivered. This is the only cost number the business will care about, and the only one that survives a model swap.

Trace every decision, not every token

The temptation, especially early, is to log everything — every token, every embedding, every intermediate string. Don't. Storage is cheap but signal-to-noise collapses fast, and you end up with a haystack and no needle.

Trace decisions. Each time the agent picks a tool, picks a branch, picks an escalation, picks to refuse — that's a span with structured attributes (which decision, which inputs, which confidence, which rationale snippet). You can always sample full token traces for a subset. You can never reconstruct decision history from raw logs after the fact.

Replays are an operator tool, not a developer tool

The single most valuable feature of an agent observability stack is the ability for an operator — not an engineer — to open a failed interaction and replay it with full context. They see the input the user gave, the context the agent retrieved, the decisions the agent made, and the output it produced. They can mark it correct, incorrect, or "should have escalated."

Those labels feed back into the eval suite. The eval suite catches regressions before they ship. The loop closes.

If the only person who can debug an agent failure is the engineer who wrote it, your operational cost will scale linearly with traffic. Build the replay tool early. It pays for itself the first week.

Don't wait for week ten

The argument for delaying observability is always the same: "we'll add it once we know what to measure." But you learn what to measure by measuring. Ship the five signals on day one with rough instrumentation. Iterate the signals as you learn the failure modes. By week ten, you'll have ten weeks of trend data instead of ten weeks of guessing.

Observability is the cheapest insurance policy in the agent stack. Buy it on day one.