Why an agent needs a different observability model
An ordinary API request usually has a clear start, a handler, and a response. An AI agent adds a loop. It selects a model, may call a tool, reads the result, and may call the model again. One request can contain several model calls, retrieval queries, tool executions, retries, and policy checks. A log line saying “request failed” does not show which branch consumed the time or budget.
Observability makes this path inspectable through correlated traces, metrics, and logs. OpenTelemetry describes traces as the path of a request and spans as operations inside it. Use one root span for the user-visible run and child spans for model inference, retrieval, tools, guardrails, and serialization. Keep model and tool names low-cardinality. Put request identifiers in trace context or logs rather than metric labels. An operator can then ask what happened in this run, how often it happens, and which workflows are affected.
The rest of this article uses a provider-neutral schema. Provider documentation remains authoritative for the exact usage fields and billing rules. The OpenTelemetry GenAI span conventions and GenAI metric conventions provide a stable vocabulary while those integrations evolve.
A trace schema that survives an agent loop
Start the root span when the application accepts the request, not at the first model. Give it an operation name such as invoke_agent and attributes for service, deployment, environment, workflow version, and a non-sensitive tenant class. Record a conversation identifier only when available and permitted. Never put a user message, full prompt, or tool payload in a metric label.
Each child span should answer one operational question. A useful minimum is:
| Span | What to record |
|---|---|
| agent.run | workflow name, workflow version, outcome, attempt count, total duration |
| gen_ai.inference | provider, requested and response model, operation, streaming flag, finish reason, token counts |
| gen_ai.retrieval | index or data-source class, query mode, result count, cache hit, duration |
| gen_ai.tool | tool name, tool type, authorization decision, timeout, result status |
| guardrail.check | policy version, decision, reason code, duration |
Use span status and an error.type value for a failed operation. Record a timestamped event for a retry with the retry number, backoff duration, and reason code. A retry is not a second root run. It is another attempt under the same logical operation, with a separate client span when a request is sent over the wire. This distinction prevents a dashboard from undercounting user requests while still showing provider attempts.
The following JSON is a shape for an exported record, not a claim about a particular vendor payload. It deliberately contains counts and codes rather than content.
{
"trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
"span_id": "00f067aa0ba902b7",
"parent_span_id": "b7ad6b7169203331",
"name": "chat model",
"kind": "CLIENT",
"status": "OK",
"attributes": {
"gen_ai.operation.name": "chat",
"gen_ai.provider.name": "provider.example",
"gen_ai.request.model": "model.example",
"gen_ai.response.model": "model.example-2026-01",
"gen_ai.usage.input_tokens": 820,
"gen_ai.usage.output_tokens": 146,
"gen_ai.response.finish_reasons": ["stop"],
"app.agent.attempt": 1
}
}
A span name should describe an operation class, not an ID or user text. Keep the semantic convention version in instrumentation metadata. If a provider reports billed tokens separately, retain both in private accounting fields and use the billed value for cost. Do not aggregate client and server instrumentation for one request without naming the layer, or totals will be counted twice.
Correlation IDs and context propagation
A trace ID is the durable link across services. A span ID identifies one operation. A request ID can remain useful for application logs, but it is not a replacement for trace context. Propagate the W3C traceparent header through your API gateway, agent service, retrieval service, and tool adapters. OpenTelemetry’s context propagation guide explains that the receiving service extracts the remote context and creates a child span. The same context can be injected into structured logs so a log search leads back to the trace.
Keep a separate random application request ID when support needs a short identifier. Store its mapping to the trace ID in logs, not a high-cardinality metric. For queues, inject context into message metadata and create a consumer span. For independent fan-out jobs, use span links when there is no single parent.
Treat incoming tracing headers as untrusted input at public boundaries. Validate their format, apply a sampling policy, and avoid forwarding internal baggage to providers or third-party tools. The OpenTelemetry guide specifically warns that baggage can carry credentials or personal data. Correlation should help an operator follow a request, not become a way to leak one.
Tool spans expose the real agent behavior
Instrument the decision and execution separately when that distinction matters. A model span shows that the model requested a tool. A tool span shows what the application actually executed. The tool span should include a stable tool.name, a tool type such as function, extension, or datastore, a policy decision, and the external operation. Add the request method or query class only when it does not reveal secrets. Do not store an access token, raw SQL parameters, document contents, or a complete URL with query data.
For each tool call, record times, timeout configuration, outcome class, retry count, and bounded result size. A timeout, business rejection, policy denial, and upstream 5xx need separate reason codes. If a tool invokes another service, propagate context. If it executes a local command, record only the command family and exit class.
The trace should also show waiting that is not a tool call. Add spans for queue delay, rate-limit sleep, circuit-breaker open time, and response streaming. Otherwise an apparently slow model call may include time spent waiting for a concurrency slot. In a multi-agent workflow, give each delegated agent a workflow name and link it to the parent trace. Avoid creating a new trace for every internal thought or state transition.
Token and cost accounting
Count usage from the provider response whenever available. Input and output, cached input, reasoning output, batch work, images, and tool units can have different rates. OpenAI’s token guide notes that tokenization varies by model and language and that the response usage object is the reliable basis for a completed request. A local tokenizer can estimate a budget, but should not replace provider usage for reconciliation.
Store a usage record per model attempt with provider, requested model, response model, token category, count, currency, pricing-table version, and an internal cost center. Calculate cost as a pure accounting step after the response:
cost = input_billable_tokens * input_price
+ cached_input_tokens * cached_input_price
+ output_billable_tokens * output_price
+ provider_units * unit_price
Use prices as dated configuration, not hard-coded exporter values. Keep raw counts and calculated amounts so corrections can be audited. Sum attempts because failed calls can consume tokens. Mark estimated or delayed charges as provisional and reconcile them with the provider report.
Expose cost dimensions that an owner can act on: workflow, model family, environment, tenant class, and outcome. Do not use user ID, prompt text, or arbitrary tool arguments as metric dimensions. A daily cost dashboard should show total amount, amount per completed run, tokens per run, retry share, and the fraction of calls using expensive models. A sudden increase in output tokens can indicate an unconstrained response, a loop, or a changed prompt even when latency is normal.
Latency distributions: p50, p95, and p99
An average hides the tail users experience during queueing or a slow tool. Record root and important span duration as histograms in seconds. Track time to first token, time between streaming chunks when relevant, and total completion time. Separate server, queue, and network time when available.
Use p50 as the typical case, p95 for a reliable “slow user” view, and p99 for rare but severe tail behavior. These are quantiles of a distribution, not three averages. Choose histogram buckets around the product’s actual objectives, such as sub-second steps through multi-minute agent runs, and keep units consistent. Prometheus’s histogram_quantile function estimates a quantile from histogram buckets. For classic histograms, aggregate by le before calculating it.
histogram_quantile(
0.95,
sum by (le, workflow) (
rate(agent_run_duration_seconds_bucket[10m])
)
)
Do not put trace IDs in metric labels. Break down latency by a small set of controlled dimensions such as workflow, model family, region, and outcome. A trace sample can then explain why a p95 point moved. Compare root latency with the sum of child spans carefully because concurrent tools overlap. Use critical-path analysis from the trace rather than adding every child duration.
Errors, retries, and rate limits
Define an error taxonomy before alerts. Distinguish validation, policy denial, authentication, rate limit, timeout, upstream error, malformed output, tool failure, and cancellation. Map provider codes into these classes and retain a bounded original code. Mark expected cancellations separately.
Every retry needs a reason, attempt number, backoff, and final disposition. Use bounded exponential backoff with jitter only when permitted. Do not retry validation, authorization, policy, or deterministic schema errors. Set deadlines for the run and each attempt. The root reports retry_count, attempt_count, and outcome while each attempt keeps its status. Retries can raise cost while success remains healthy.
Record fallback model selection as an explicit event or span. Dashboards should answer whether a fallback was caused by rate limits, latency, safety policy, or a planned capability check. Monitor malformed tool arguments and schema-repair loops separately. If a repair loop is allowed, cap iterations and emit a terminal loop_limit reason when the cap is reached.
Privacy, redaction, and sampling
Prompt and tool data can contain personal, confidential, or security-sensitive information. The safest default is metadata, token counts, hashes, and reason codes, with content out of telemetry. If debugging needs examples, use a separately controlled store with consent, short retention, encryption, access logs, and field-level redaction. Redact before export.
Use attribute allowlists. Remove authorization headers, cookies, API keys, contact data, account numbers, raw URLs with query strings, and document text. Hashing is not automatically anonymous. Keep prompt fingerprints separate and document who can join them. Test redaction with realistic secrets and multilingual personal data.
Sampling limits volume, but it must not hide incidents. Use parent-based sampling for trace consistency. Consider tail sampling in the Collector to retain errors, timeouts, high-cost runs, and slow traces while sampling ordinary successful runs. The OpenTelemetry sampling specification distinguishes recording from export, so a local sampler can also avoid expensive attribute construction. Keep metrics unsampled and use traces for exemplars and investigation.
OpenTelemetry, Grafana, and Sentry together
Instrument the agent with OpenTelemetry APIs and semantic conventions, send OTLP to a Collector, and let it apply batching, memory limits, redaction, sampling, and routing. Export traces, metrics, and logs to their backends with consistent service, version, environment, region, and deployment attributes. Validate one complete staging trace before production sampling.
Grafana provides shared operational views. Build panels for volume, success ratio, p50/p95/p99 root latency, first-token time, tokens, cost, retries, tool duration, and provider status. Link panels to trace searches and runbooks. Grafana’s alert rule documentation covers queries, conditions, evaluation, and notifications. Sentry adds issue grouping, error context, and trace inspection. Its trace API exposes the spans and errors in one trace. Send only redacted data and make sampling explicit.
SLOs and alerts that operators can use
An SLO should represent a user-visible promise, not collector health. Define availability as completed runs without a classified server, provider, or tool error. Define latency as root runs below a chosen threshold. If cost is a product constraint, track a separate budget SLI.
Choose targets from a measured baseline and product requirement. Use separate SLOs for workflows with different expectations. Report error budget over a long window and keep a short deployment view. Grafana’s SLO documentation describes SLI queries, budget consumption, and fast- and slow-burn alerts. Page only when a fast burn is actionable. Use a ticket for a slow trend.
Useful alerts include a sustained increase in failed root runs, a rapid error-budget burn, p95 latency above the contract, a provider rate-limit surge, a growing retry ratio, missing usage records, an unexpected cost rate, and a stalled queue. Include workflow, region, deployment, current value, threshold, trace search link, owner, and runbook in the alert. Configure pending periods to avoid paging on one sample and group notifications by service and severity. A dashboard is better than an alert when no immediate action exists.
A troubleshooting path from symptom to cause
Begin with the SLO or user report and select a representative trace. Check root children and context at gateway, queue, and tool boundaries. If spans are missing, inspect exporter health, sampling, and context injection before application code.
For slow runs, compare queue delay, first-token time, output duration, retrieval, and tool critical paths. High p99 with normal p50 usually points to a tail dependency, concurrency limit, or retry. A p50 shift suggests a deployment, model, prompt, or region change. For a cost spike, group by response model, token category, workflow version, and attempts, then reconcile usage with the provider report.
For errors, start at the first failed span, not the wrapper exception. Check status, error.type, provider code, timeout budget, and retry events. Separate tool rejection, provider outage, malformed response, and parser bug. Ensure redaction kept a safe diagnostic code. Count expected policy blocks as product outcomes and alert only on an unexpected rate change.
Keep synthetic requests with non-sensitive deterministic fixtures. Run them after instrumentation, model, prompt, or routing changes and verify that traces, metrics, token records, and errors share one correlation ID. For architecture context, see production AI agent architecture, AI agent evaluations, prompt injection and MCP security, and hybrid RAG with pgvector. Those topics change which spans exist and which SLO is acceptable, while observability remains the evidence layer.