Agent Observability Guide

Capture model, tool, cost, latency, state, approval, error, and outcome signals without recording secrets or excessive user content.

The result you're building

A privacy-bounded telemetry system that reconstructs each agent run across model, tool, approval, cost, latency, state, error, and outcome without storing raw secrets or unnecessary user content.

Use this guide when

• Diagnosing agent failures, loops, wrong tool calls, cost spikes, or approval disputes.
• Defining production SLOs and incident evidence.

Do not use it as a substitute for

• Do not log full prompts, documents, tool responses, tokens, cookies, or wallet signatures by default.
• Do not call a run successful because the model returned text.

Before you change anything

• Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
• Run/task/tool identifiers and state model.
• Data classification, retention, access, and redaction policy.
• Outcome/SLO/cost definitions and incident questions.
• Trace/log/metric storage and deletion/export requirements.

Understand the system before fixing it

Events reconstruct; metrics alert
Traces/events answer what happened; aggregate metrics detect rate changes. Both need stable IDs and definitions.

Outcome is outside model completion
A tool may fail after a polished answer, or a payment may settle without delivery. Record verified business outcome.

Redaction is a schema design
Allowlist safe metadata and hashes/references rather than collecting everything then trying to scrub.

Evidence-to-decision map

Step-by-step procedure

Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.

Step 01 — Define decisions and SLOs

Why: Collecting telemetry without questions creates privacy cost.

Do: List incidents/decisions: success, unsafe action, latency, paid failure, cost, duplicate, approval, and quality; define numerator/denominator/window.

Read the result: Every event field must support a stated use.

Next: Set alert and retention thresholds.

Step 02 — Create run/state/event schema

Why: Free-text logs cannot reconstruct concurrency and retries.

Do: Use stable run/task/step/tool/attempt/business IDs; timestamp state transitions; record model/prompt/tool/schema/policy versions.

Read the result: One timeline can show planned, requested, approved, executed, verified, failed, canceled.

Next: Distinguish retry attempts from logical operation.

Step 03 — Allowlist safe telemetry

Why: Default capture leaks content.

Do: Store categories, sizes, hashes/references, counts, statuses, latency, cost, and sanitized error codes; redact/drop secrets before sink.

Read the result: Raw content requires exceptional justified path with access/expiry.

Next: Canary-test every error/tool path.

Step 04 — Join model, tool, and outcome

Why: Separate dashboards hide causality.

Do: Propagate correlation IDs to tool/backend/payment and emit postcondition/reconciliation result.

Read the result: Agent claim and external verified outcome remain separate fields.

Next: Track duplicates and ambiguous states.

Step 05 — Build alerts and investigation views

Why: Operators need actionable deviation.

Do: Alert on unsafe/refusal regression, tool/paid failure, p95 latency, cost/run, loop depth, duplicates, auth errors, and redaction hits with baseline-aware thresholds.

Read the result: Avoid alerting on sensitive content.

Next: Create run timeline and aggregate trend.

Step 06 — Test privacy and failure behavior

Why: Telemetry itself can fail or become attack path.

Do: Inject canary secrets, oversized/malicious tool output, sink outage, clock skew, duplicate events, and deletion request.

Read the result: Agent continues safely or fails per policy; secrets absent; retention/deletion work.

Next: Review access audit.

Worked example

Evidence collected

• Aggregate cost lacks step/tool breakdown.
• New structured-output validator retries each schema failure three times.
• A model update increases one invalid field from 2% to 25%.
• Business outcome is recorded separately and is stable.

Decision: Cost spike is a repair loop caused by model/schema compatibility, not more user demand.

Actions taken

• Added per-attempt model/schema/error/cost events.
• Capped repairs and routed repeated failure to fallback/schema fix.
• Created alert on validation-repair rate and cost per verified outcome.

Why this example matters: Outcome-linked telemetry found a hidden loop that top-line success concealed.

Verify, recover, and hand off

Completion tests

• A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
• One run timeline joins model/tool/approval/backend/outcome.
• Logical operations are distinct from retry attempts.
• Cost/latency/error/success definitions are documented.
• Canary secrets and raw sensitive content are absent by default.
• Alerts fire on tested deviations.
• Retention/access/deletion and sink failure are tested.

Rollback or safe recovery

• Disable newly added high-risk fields while preserving safe core metrics.
• Revert event schema/dashboard version and replay sanitized events if compatible.
• Purge/rotate access after telemetry leak and follow incident policy.

If the expected result does not appear

Reusable handoff record

• Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
• Observability questions/SLO definitions.
• Event/state/correlation/version schema.
• Safe-field/redaction/retention/access policy.
• Dashboards/alerts and verified outcome linkage.
• Privacy/failure test report and incident runbook.

Agent delivery contract

Required inputs

Returned output

Agent refusal and escalation rules

• Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
• Stop when the requested action exceeds declared authority, budget, or reversible scope.
• Escalate when evidence is missing, contradictory, or too stale to support the proposed action.

Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.

Official reference starting points

• https://opentelemetry.io/docs/concepts/observability-primer/
• https://opentelemetry.io/docs/specs/otel/trace/