SAYLOR The result you are building
Finished result: A versioned tool catalog that measurably improves correct tool selection and first-call input validity while reducing overlap, ambiguity, excessive arguments, and unsafe defaults.
Use this guide when
- Agents select the wrong tool, omit fields, hallucinate enums, or need repeated correction.
- A tool catalog grew organically and now overlaps.
Do not use it as a substitute for
- Do not optimize descriptions from anecdotes alone; keep an evaluation set and baseline.
- Do not place authorization or safety guarantees only in descriptions.
Before you change anything
- Current tool schemas/descriptions and call logs with secrets removed.
- Representative user intents, including non-use and adversarial cases.
- Validation errors, wrong-tool rate, retries, latency, and cost baseline.
- Security/authority differences between tools.
Understand the system before fixing it
Names should encode action and object. get_invoice is easier to distinguish than fetch_data. Names remain short but outcome-specific.
Descriptions need decision boundaries. State when to use, when not to use, required evidence, side effects, and returned result.
Schema complexity becomes model error. Deep unions, free-form maps, duplicated fields, and implicit defaults increase invalid calls.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Wrong tool among similar names | Catalog overlap | Run paired-intent selection eval | Merge/split/rename and add explicit non-use boundaries. |
| Missing required field | Description/schema | Inspect which intent lacks value | Make field required only if user/tool can supply it; add acquisition guidance. |
| Invalid free text choice | Type design | Observed values vs allowed set | Use enum plus meaning; reject unknown. |
| Valid call causes unsafe default | Default/authority | Omit optional consequential field | Require explicit value/confirmation; no dangerous default. |
Step-by-step procedure
01 Build an intent and error baseline
Why: Optimization needs measurable targets. Do: Sample real intents, label correct tool/arguments/refusal, and compute selection, validity, retries, cost, and failure classes. Read the result: Frequent overlap/field errors identify priority. Next: Freeze holdout eval set.
02 Remove catalog overlap
Why: Descriptions cannot fully rescue duplicate capabilities. Do: For each pair document unique outcome, authority, and non-use case; merge redundant read tools or split mixed actions. Read the result: Every intent should have one best tool or an explicit ambiguity question. Next: Keep dangerous tools visibly distinct.
03 Rewrite name and description
Why: The agent must decide before seeing parameter details. Do: Use action-object name; first sentence states exact result; then when/not when, side effect, and constraints. Read the result: Avoid marketing language and implementation trivia. Next: Run blind selection eval.
04 Simplify parameters
Why: Fewer explicit fields outperform nested optional blobs. Do: Require true necessities, use enums/bounds/formats, separate user values from server context, reject unknown properties, and include valid/invalid examples. Read the result: Do not default recipients, amounts, deletion, network, or production environment. Next: Version breaking changes.
05 Align errors and outputs
Why: Agents retry blindly when errors do not identify corrective fields. Do: Return stable validation paths/codes and a bounded typed result with provenance and next allowed action. Read the result: Errors must never echo secrets or authorize broadened behavior. Next: Add repair-call evals.
06 Compare and release
Why: A prettier schema may perform worse. Do: Run baseline and candidate on held-out normal/boundary/adversarial intents; compare selection, validity, unsafe-call, latency, and tokens. Read the result: Release only material improvement without safety regression. Next: Keep previous schema version.
Worked example
Starting problem: Tools search, find, and lookup all claim to retrieve customer data.
Evidence collected
- Wrong-tool rate is 31%.
- Only one supports fuzzy text; one requires exact ID; one searches invoices, not customers.
- Descriptions share the phrase 'find data'.
Decision: The catalog communicates implementation names instead of distinct outcomes.
Actions taken
- Renamed to
search_customers,get_customer_by_id, andsearch_invoices. - Added when/not-when and examples; made IDs formatted strings.
- Re-ran held-out selection and invalid-input tests.
Proof of completion: Wrong-tool rate falls below target, first-call validity improves, and no new authority/unsafe-default failure appears.
Why this example matters: Optimization is demonstrated by evaluation results, not subjective wording.
Verify, recover, and hand off
Completion tests
- Each labeled intent has one correct tool or defined clarification.
- Required/enums/bounds reject bad inputs with field-level errors.
- Dangerous actions have no implicit defaults.
- Held-out selection and validity beat baseline.
- Unsafe-call/refusal performance does not regress.
- Schema version/migration is documented.
Rollback or safe recovery
- Route clients to prior schema version.
- Restore previous names/descriptions while keeping evaluation evidence.
- Disable a tool whose new schema changes authority unexpectedly.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Selection improves but validity drops | Description and parameter contract disagree | Align required fields/examples or split intent. |
| Token count grows sharply | Descriptions over-explain shared rules | Move common policy to server/catalog docs; keep tool boundary concise. |
| Agent invents enum | Values or meaning unclear | Use short explicit enums with descriptions/examples. |
| Legacy client breaks | Breaking schema rename lacks versioning | Maintain alias/version and migration period without duplicate agent-visible overlap. |
Reusable handoff record
- Baseline catalog and labeled evaluation set.
- Overlap/authority matrix.
- Candidate schemas and examples.
- Before/after metrics including safety.
- Version/migration/rollback decision.
For agents
This guide also defines a structured diagnose/propose/execute contract for building an agent-facing tool around this workflow (a design reference, not a live endpoint on this site today):
Required inputs: context (versioned environment/target/outcome), evidence (timestamped observations and sanitized results), constraints (authority/risk/downtime/budget/reversibility limits), success (observable acceptance checks).
Returned output: diagnosis (likely layer, evidence, alternatives, confidence), plan (ordered actions with risk and expected evidence), verification (pass/fail checks), handoff (sanitized evidence record, remaining risks, rollback state).
Refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: score confidence from the number and quality of independent observations, not from how familiar the error looks.
References
- https://json-schema.org/understanding-json-schema/
- https://modelcontextprotocol.io/specification/2026-07-28/server/tools