SAYLOR Turn an agent specification into repeatable normal, boundary, adversarial, permission, cost, and regression tests with clear pass criteria.
The result you're building
A versioned evaluation suite with representative, boundary, adversarial, permission, cost, and regression cases; deterministic graders where possible; human rubrics where necessary; baselines; and release gates tied to real outcomes.
Use this guide when
- Before changing an agent model, prompt, tools, schemas, permissions, or workflow.
- Converting production failures into permanent regression cases.
Do not use it as a substitute for
- Do not grade style while ignoring unsafe side effects or wrong factual outcomes.
- Do not reuse only the examples used to tune the prompt.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Agent specification, tools, policies, and success/refusal criteria.
- Real task distribution and sanitized failure examples.
- Cost/latency budgets and allowed nondeterminism.
- Safe sandbox/mocks and grading owners.
Understand the system before fixing it
Evaluate behavior and state, not prose
Check tool calls, arguments, side effects, final external state, citations, spend, and refusals.
Critical cases need zero-tolerance gates
Average scores can hide a single catastrophic action.
A held-out set prevents prompt overfitting
Keep tuning examples separate from release/regression evaluation.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Happy path passes, edge fails | Coverage | Boundary partitions for every input/tool | Add empty/max/invalid/ambiguous/timeouts. |
| Answer sounds right, action wrong | Grader | Inspect external state/tool trace | Use deterministic outcome grader. |
| Flaky score | Nondeterminism | Repeat seeds/runs and confidence interval | Define acceptable pass probability/variance. |
| Adversarial bypass | Safety | Trace policy/tool boundary | Block release; fix independent control and preserve case. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Translate spec into observable claims
Why: Vague 'works well' cannot be tested.
Do: List correct tool/arguments, outcome, required evidence, refusal/escalation, budget, latency, and prohibited effects per task.
Read the result: Each claim gets a grader and severity.
Next: Critical safety gates are binary.
Step 02 — Partition case space
Why: Production failures live at boundaries.
Do: Create normal, empty, max, malformed, ambiguous, conflicting, stale, timeout, duplicate, unavailable dependency, permission, injection, secret, cross-tenant, and cost cases.
Read the result: Sample distribution and rare critical risks separately.
Next: Keep tuning/holdout sets.
Step 03 — Build controlled fixtures
Why: Live systems make evaluation unsafe and irreproducible.
Do: Use mocks/sandbox/test accounts, fixed data versions, deterministic IDs/time, and cleanup/reconciliation.
Read the result: No production message/payment/delete can occur.
Next: Record fixture/schema versions.
Step 04 — Choose graders
Why: LLM judges alone can reward fluent wrong answers.
Do: Prefer exact schema, tool trace, database/state, citation, arithmetic, and policy graders; use rubric/human review for nuanced quality with calibration.
Read the result: Grade claimed success against actual outcome.
Next: Capture reason and evidence.
Step 05 — Run baseline/candidate with repeats
Why: One run hides variance.
Do: Use same held-out cases, multiple runs where nondeterministic, and compare critical pass, task success, wrong-tool, invalid-call, refusal, cost, and latency.
Read the result: No candidate ships with critical regression.
Next: Analyze failure clusters, not just average.
Step 06 — Convert incidents to regression
Why: Tests must grow with reality.
Do: Sanitize and minimize each production failure, add expected behavior, reproduce against old version, and require pass before release.
Read the result: Version suite and thresholds.
Next: Review for stale cases/data.
Worked example
Evidence collected
- Overall score averages safety and style.
- Email tool test uses mock but grader checks only final text.
- Tool trace shows unauthorized send.
- Injection case is one of 200.
Decision: Average metric hides a critical authority failure.
Actions taken
- Made unauthorized side effect a zero-tolerance gate.
- Added deterministic tool/state grader and injection variants.
- Fixed tool approval/egress boundary and reran held-out suite.
Why this example matters: Severity-aware gates prevent cosmetic gains from buying catastrophic risk.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Every spec/policy claim maps to observable grader.
- Critical side effects use deterministic state/tool checks.
- Normal/boundary/adversarial/permission/cost cases exist.
- Held-out and repeated runs quantify variance.
- Baseline/candidate report includes failures and severity.
- Production incidents become minimized regression tests.
Rollback or safe recovery
- Route to last passing model/prompt/tool/schema bundle.
- Disable changed high-impact tool/policy independently.
- Preserve failed candidate results and test version for diagnosis.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Judge disagrees with humans | Rubric/calibration weak. | Calibrate on labeled set or use deterministic evidence. |
| Tests pass; production fails | Distribution/fixture gap. | Minimize incident and expand cases/state realism. |
| Suite too expensive | No tiering/caching. | Run fast critical smoke on every change; full suite on release. |
| Flakiness masks regression | Uncontrolled data/time/model variance. | Pin fixtures, repeat, and report confidence/variance. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Observable requirements and severity map.
- Versioned cases/fixtures/graders.
- Baseline/candidate repeated results.
- Critical release gates and cost/latency thresholds.
- Failure clusters and regression additions.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points