SAYLOR The result you are building
Finished result: A privacy-bounded telemetry system that reconstructs each agent run across model, tool, approval, cost, latency, state, error, and outcome without storing raw secrets or unnecessary user content.
Use this guide when
- Diagnosing agent failures, loops, wrong tool calls, cost spikes, or approval disputes.
- Defining production SLOs and incident evidence.
Do not use it as a substitute for
- Do not log full prompts, documents, tool responses, tokens, cookies, or wallet signatures by default.
- Do not call a run successful because the model returned text.
Before you change anything
- Run/task/tool identifiers and state model.
- Data classification, retention, access, and redaction policy.
- Outcome/SLO/cost definitions and incident questions.
- Trace/log/metric storage and deletion/export requirements.
Understand the system before fixing it
Events reconstruct; metrics alert. Traces/events answer what happened; aggregate metrics detect rate changes. Both need stable IDs and definitions.
Outcome is outside model completion. A tool may fail after a polished answer, or a payment may settle without delivery. Record verified business outcome.
Redaction is a schema design. Allowlist safe metadata and hashes/references rather than collecting everything then trying to scrub.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| High token cost | Planning/context/loop | Break cost by step/model/tool/retry | Find repeated context, repair loops, or wrong routing. |
| Tool errors without cause | Missing correlation | Join run/tool/backend request IDs | Propagate IDs and stable error codes. |
| Success text but failed task | Outcome gap | Compare claimed vs verified side effect | Add postcondition and reconciliation event. |
| Incident cannot be reproduced | Version/state gap | Model/prompt/tool/schema/policy/data-version fields | Record immutable versions and safe input references. |
Step-by-step procedure
01 Define decisions and SLOs
Why: Collecting telemetry without questions creates privacy cost. Do: List incidents/decisions: success, unsafe action, latency, paid failure, cost, duplicate, approval, and quality; define numerator/denominator/window. Read the result: Every event field must support a stated use. Next: Set alert and retention thresholds.
02 Create run/state/event schema
Why: Free-text logs cannot reconstruct concurrency and retries. Do: Use stable run/task/step/tool/attempt/business IDs; timestamp state transitions; record model/prompt/tool/schema/policy versions. Read the result: One timeline can show planned, requested, approved, executed, verified, failed, canceled. Next: Distinguish retry attempts from logical operation.
03 Allowlist safe telemetry
Why: Default capture leaks content. Do: Store categories, sizes, hashes/references, counts, statuses, latency, cost, and sanitized error codes; redact/drop secrets before sink. Read the result: Raw content requires exceptional justified path with access/expiry. Next: Canary-test every error/tool path.
04 Join model, tool, and outcome
Why: Separate dashboards hide causality. Do: Propagate correlation IDs to tool/backend/payment and emit postcondition/reconciliation result. Read the result: Agent claim and external verified outcome remain separate fields. Next: Track duplicates and ambiguous states.
05 Build alerts and investigation views
Why: Operators need actionable deviation. Do: Alert on unsafe/refusal regression, tool/paid failure, p95 latency, cost/run, loop depth, duplicates, auth errors, and redaction hits with baseline-aware thresholds. Read the result: Avoid alerting on sensitive content. Next: Create run timeline and aggregate trend.
06 Test privacy and failure behavior
Why: Telemetry itself can fail or become attack path. Do: Inject canary secrets, oversized/malicious tool output, sink outage, clock skew, duplicate events, and deletion request. Read the result: Agent continues safely or fails per policy; secrets absent; retention/deletion work. Next: Review access audit.
Worked example
Starting problem: Agent costs triple while completion rate looks unchanged.
Evidence collected
- Aggregate cost lacks step/tool breakdown.
- New structured-output validator retries each schema failure three times.
- A model update increases one invalid field from 2% to 25%.
- Business outcome is recorded separately and is stable.
Decision: Cost spike is a repair loop caused by model/schema compatibility, not more user demand.
Actions taken
- Added per-attempt model/schema/error/cost events.
- Capped repairs and routed repeated failure to fallback/schema fix.
- Created alert on validation-repair rate and cost per verified outcome.
Proof of completion: Repair rate returns to threshold; cost per verified outcome returns near baseline; no sensitive output was added to telemetry.
Why this example matters: Outcome-linked telemetry found a hidden loop that top-line success concealed.
Verify, recover, and hand off
Completion tests
- One run timeline joins model/tool/approval/backend/outcome.
- Logical operations are distinct from retry attempts.
- Cost/latency/error/success definitions are documented.
- Canary secrets and raw sensitive content are absent by default.
- Alerts fire on tested deviations.
- Retention/access/deletion and sink failure are tested.
Rollback or safe recovery
- Disable newly added high-risk fields while preserving safe core metrics.
- Revert event schema/dashboard version and replay sanitized events if compatible.
- Purge/rotate access after telemetry leak and follow incident policy.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Trace IDs break at tool | Context not propagated | Pass explicit correlation metadata and map backend request ID. |
| High cardinality cost | Raw IDs/labels used in metrics | Keep IDs in traces/logs; aggregate bounded labels in metrics. |
| Logs still leak secrets | Unhandled exception/upstream body path | Central allowlist serialization and canary tests. |
| Success metric inflated | Counts model completion not verified outcome | Use postcondition/business reconciliation denominator. |
Reusable handoff record
- Observability questions/SLO definitions.
- Event/state/correlation/version schema.
- Safe-field/redaction/retention/access policy.
- Dashboards/alerts and verified outcome linkage.
- Privacy/failure test report and incident runbook.
For agents
This guide also defines a structured diagnose/propose/execute contract for building an agent-facing tool around this workflow (a design reference, not a live endpoint on this site today):
Required inputs: context, evidence, constraints, success. Returned output: diagnosis, plan, verification, handoff.
Refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: score confidence from the number and quality of independent observations, not from how familiar the error looks.
References
- https://opentelemetry.io/docs/concepts/observability-primer/
- https://opentelemetry.io/docs/specs/otel/trace/