SAYLOR Measure and control model, tool, data, payment, retry, and human-review cost while preserving the quality needed for the task.
The result you're building
A per-task budget policy and scorecard that predicts maximum spend and deadline, records actual contribution by step, degrades safely, and stops workflows whose expected value, quality, or remaining budget is insufficient.
Use this guide when
- Agents call multiple models, APIs, paid x402 resources, browsers, or people.
- Latency or cost varies significantly by route and retry.
- You need defensible routing and price decisions instead of selecting the cheapest headline model.
Do not use it as a substitute for
- Optimizing token price while ignoring tool, failure, review, and opportunity cost.
- Allowing retries or parallel fan-out to spend an unbounded budget.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Task classes, required quality, deadline, and economic value.
- Model/tool/data prices, quotas, p50/p95 latency, and failure rates.
- Prompt/input/output sizes and cache behavior.
- Retry, parallelism, human review, and paid-failure policy.
- Historical quality, abstention, correction, and end-to-end success by route.
Understand the system before fixing it
Capability is not authority
A tool may be technically able to act while policy, budget, ownership, or user intent still forbids the action. Keep execution permission explicit and revocable.
Deterministic boundaries contain probabilistic reasoning
Models may propose plans, but schemas, allowlists, budgets, approvals, idempotency, and postconditions must decide what can actually happen.
Unit economics are end to end
Include model tokens, tools, data, network, payment fees, failed attempts, review, storage, support, and refunded or unusable outputs.
Cheapest valid route beats cheapest call
A low-cost call that triggers retries, correction, or bad action can be more expensive than a stronger route that succeeds once.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Token spend is low but task cost is high | Tool/review cost | Allocate every cost to task and step | Non-model calls, retries, or human correction dominate. |
| p50 fast but users time out | Tail latency | Trace p95/p99 critical path | A slow dependency or sequential chain controls completion. |
| Cache saves money but answers stale | Freshness policy | Compare cache age to task requirement | Cost optimization violates the answer contract. |
| Parallel agents exceed cap | Fan-out | Calculate worst-case branch spend | Budget enforcement happens after calls rather than before reservation. |
| Cheap model produces more fixes | Quality routing | Compare end-to-end success and correction cost | Per-call price is the wrong objective. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Define task value and minimum quality
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Classify tasks by consequence, required accuracy, freshness, deadline, privacy, and maximum economically rational spend.
Read the result: Each task class has a quality floor and hard spend/deadline cap.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Build the full cost model
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Record model input/output, tool/API/x402, compute, storage, network, payment, retries, failed paid calls, review, and support cost per successful result.
Read the result: Historical task totals reconcile to provider and internal records.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Trace the latency critical path
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Measure queue, model, tool, network, approval, and retry spans. Report p50, p95, p99, timeout, and deadline miss by route.
Read the result: The slowest dependency and sequential waits are visible.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Reserve budget before execution
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Estimate worst-case next-step and branch cost, reserve against task/daily caps, and reject or request approval before exceeding them.
Read the result: Parallel calls cannot overrun the cap through races.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Route by expected total value
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Choose models and tools using quality, success, freshness, latency, privacy, and total correction cost. Use cheaper routes only when acceptance tests still pass.
Read the result: Routing decisions improve cost per accepted result, not merely cost per call.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Design bounded degradation
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: When budget or time tightens, reduce optional enrichment, return partial with explicit omissions, or stop. Never silently lower required safety checks.
Read the result: Degraded output is labeled, schema-valid, and within the declared quality floor.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Review variance and kill waste
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Alert on cost, latency, retry, cache, quality, and margin drift. Disable routes whose accepted-result economics fail policy.
Read the result: Scorecards show forecast vs actual and a tested stop control.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Task classes, required quality, deadline, and economic value.
- Model/tool/data prices, quotas, p50/p95 latency, and failure rates.
- Prompt/input/output sizes and cache behavior.
- Retry, parallelism, human review, and paid-failure policy.
- Historical quality, abstention, correction, and end-to-end success by route.
Acceptance scoreboard
- Task classes declare quality, freshness, privacy, deadline, and spend caps.
- Actual end-to-end cost reconciles across every provider and internal step.
- Worst-case parallel and retry spend is reserved before calls.
- Routing is evaluated by accepted-result quality, cost, and latency.
- Degradation never removes mandatory safety or authority checks.
- Alerts and kill switches respond to margin, cost, latency, and quality drift.
Minimum handoff record
- Versioned agent cost, latency, and quality budgeting scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- Model cost is 12% of the task total.
- Search/tool cost is 63%; review and retries are 25%.
- Parallel branches reserve no budget.
- Only six results contribute to the accepted answer.
Decision: The routing objective is wrong. Bound discovery, rank before enrichment, reserve branch budgets, and measure cost per accepted result.
Actions taken
- Added a two-stage search-and-rank path.
- Capped and reserved fan-out spend.
- Stopped low-value branches early.
- Tracked evidence contribution and deadline at each stage.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Task classes declare quality, freshness, privacy, deadline, and spend caps.
- Actual end-to-end cost reconciles across every provider and internal step.
- Worst-case parallel and retry spend is reserved before calls.
- Routing is evaluated by accepted-result quality, cost, and latency.
- Degradation never removes mandatory safety or authority checks.
- Alerts and kill switches respond to margin, cost, latency, and quality drift.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Token spend is low but task cost is high | Non-model calls, retries, or human correction dominate. | Allocate every cost to task and step |
| p50 fast but users time out | A slow dependency or sequential chain controls completion. | Trace p95/p99 critical path |
| Cache saves money but answers stale | Cost optimization violates the answer contract. | Compare cache age to task requirement |
| Parallel agents exceed cap | Budget enforcement happens after calls rather than before reservation. | Calculate worst-case branch spend |
Reusable handoff record
- Versioned agent cost, latency, and quality budgeting scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points