SAYLOR Rewrite tool names, descriptions, parameters, enums, and examples so agents choose the correct tool and provide valid inputs on the first attempt.
The result you're building
A versioned tool catalog that measurably improves correct tool selection and first-call input validity while reducing overlap, ambiguity, excessive arguments, and unsafe defaults.
Use this guide when
- Agents select the wrong tool, omit fields, hallucinate enums, or need repeated correction.
- A tool catalog grew organically and now overlaps.
Do not use it as a substitute for
- Do not optimize descriptions from anecdotes alone; keep an evaluation set and baseline.
- Do not place authorization or safety guarantees only in descriptions.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Current tool schemas/descriptions and call logs with secrets removed.
- Representative user intents, including non-use and adversarial cases.
- Validation errors, wrong-tool rate, retries, latency, and cost baseline.
- Security/authority differences between tools.
Understand the system before fixing it
Names should encode action and objectget_invoice is easier to distinguish than fetch_data. Names remain short but outcome-specific.
Descriptions need decision boundaries
State when to use, when not to use, required evidence, side effects, and returned result.
Schema complexity becomes model error
Deep unions, free-form maps, duplicated fields, and implicit defaults increase invalid calls.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Wrong tool among similar names | Catalog overlap | Run paired-intent selection eval | Merge/split/rename and add explicit non-use boundaries. |
| Missing required field | Description/schema | Inspect which intent lacks value | Make field required only if user/tool can supply it; add acquisition guidance. |
| Invalid free text choice | Type design | Observed values vs allowed set | Use enum plus meaning; reject unknown. |
| Valid call causes unsafe default | Default/authority | Omit optional consequential field | Require explicit value/confirmation; no dangerous default. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Build an intent and error baseline
Why: Optimization needs measurable targets.
Do: Sample real intents, label correct tool/arguments/refusal, and compute selection, validity, retries, cost, and failure classes.
Read the result: Frequent overlap/field errors identify priority.
Next: Freeze holdout eval set.
Step 02 — Remove catalog overlap
Why: Descriptions cannot fully rescue duplicate capabilities.
Do: For each pair document unique outcome, authority, and non-use case; merge redundant read tools or split mixed actions.
Read the result: Every intent should have one best tool or an explicit ambiguity question.
Next: Keep dangerous tools visibly distinct.
Step 03 — Rewrite name and description
Why: The agent must decide before seeing parameter details.
Do: Use action-object name; first sentence states exact result; then when/not when, side effect, and constraints.
Read the result: Avoid marketing language and implementation trivia.
Next: Run blind selection eval.
Step 04 — Simplify parameters
Why: Fewer explicit fields outperform nested optional blobs.
Do: Require true necessities, use enums/bounds/formats, separate user values from server context, reject unknown properties, and include valid/invalid examples.
Read the result: Do not default recipients, amounts, deletion, network, or production environment.
Next: Version breaking changes.
Step 05 — Align errors and outputs
Why: Agents retry blindly when errors do not identify corrective fields.
Do: Return stable validation paths/codes and a bounded typed result with provenance and next allowed action.
Read the result: Errors must never echo secrets or authorize broadened behavior.
Next: Add repair-call evals.
Step 06 — Compare and release
Why: A prettier schema may perform worse.
Do: Run baseline and candidate on held-out normal/boundary/adversarial intents; compare selection, validity, unsafe-call, latency, and tokens.
Read the result: Release only material improvement without safety regression.
Next: Keep previous schema version.
Worked example
search, find, and lookup all claim to retrieve customer data.Evidence collected
- Wrong-tool rate is 31%.
- Only one supports fuzzy text; one requires exact ID; one searches invoices, not customers.
- Descriptions share the phrase 'find data'.
Decision: The catalog communicates implementation names instead of distinct outcomes.
Actions taken
- Renamed to
search_customers,get_customer_by_id, andsearch_invoices. - Added when/not-when and examples; made IDs formatted strings.
- Re-ran held-out selection and invalid-input tests.
Why this example matters: Optimization is demonstrated by evaluation results, not subjective wording.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Each labeled intent has one correct tool or defined clarification.
- Required/enums/bounds reject bad inputs with field-level errors.
- Dangerous actions have no implicit defaults.
- Held-out selection and validity beat baseline.
- Unsafe-call/refusal performance does not regress.
- Schema version/migration is documented.
Rollback or safe recovery
- Route clients to prior schema version.
- Restore previous names/descriptions while keeping evaluation evidence.
- Disable a tool whose new schema changes authority unexpectedly.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Selection improves but validity drops | Description and parameter contract disagree. | Align required fields/examples or split intent. |
| Token count grows sharply | Descriptions over-explain shared rules. | Move common policy to server/catalog docs; keep tool boundary concise. |
| Agent invents enum | Values or meaning unclear. | Use short explicit enums with descriptions/examples. |
| Legacy client breaks | Breaking schema rename lacks versioning. | Maintain alias/version and migration period without duplicate agent-visible overlap. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Baseline catalog and labeled evaluation set.
- Overlap/authority matrix.
- Candidate schemas and examples.
- Before/after metrics including safety.
- Version/migration/rollback decision.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points