SAYLOR Diagnose invalid JSON, schema mismatches, truncation, unsupported unions, and repair loops without silently accepting corrupted data.
The result you're building
A deterministic pipeline that validates model output against a supported schema, distinguishes syntax/truncation/schema/semantic failures, retries with bounded repair, and never silently accepts corrupted or invented fields.
Use this guide when
- Model JSON is invalid, truncated, schema-incompatible, or intermittently missing fields.
- Downstream automation needs reliable typed records.
Do not use it as a substitute for
- Do not regex-patch arbitrary JSON into validity and call it correct.
- Do not coerce missing high-impact facts or drop validation errors silently.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Exact schema and provider/model structured-output capabilities.
- Raw response, finish/stop reason, token limits, and validation errors.
- Expected semantic invariants and acceptable null/reject behavior.
- Retry/cost/latency budget and side-effect boundary.
Understand the system before fixing it
Valid JSON is weaker than valid data
Syntax can pass while IDs, dates, totals, relationships, or evidence are wrong.
Truncation is not a schema-repair problem
Finish reason and incomplete tail require smaller output/pagination or more budget, not brace insertion.
Provider-supported schema subset matters
Unions, recursion, additional properties, defaults, and formats may behave differently; test the actual provider/model.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Parser error at end | Truncation/syntax | Finish reason and raw tail | Increase/split output only after confirming limit; never append guessed content. |
| Type/required error | Schema | JSON Pointer validation errors | Repair exact fields or redesign schema. |
| Schema passes; totals impossible | Semantic | Cross-field invariant/calculation | Reject and re-extract from source; schema alone insufficient. |
| Repair loop changes facts | Unsafe repair | Diff raw vs repaired values/provenance | Repairs may format only; factual change requires source evidence. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Capture raw response metadata
Why: Repair destroys diagnosis if raw bytes and finish reason are lost.
Do: Store sanitized raw output, model/version, request/schema hash, finish reason, usage, and timestamp.
Read the result: Truncated/refused/tool-call outputs take different branches.
Next: Do not log secrets/full sensitive prompts.
Step 02 — Parse without heuristic mutation
Why: Parser position identifies syntax class.
Do: Attempt strict JSON parse once; record error offset/context.
Read the result: If truncated, reduce/split/regenerate; if wrappers/fences, fix generation contract rather than global regex.
Next: Never execute partial data.
Step 03 — Validate exact schema
Why: Loose validation lets unknown/mistyped data through.
Do: Use pinned JSON Schema validator, reject unknown fields where intended, and return JSON Pointer errors.
Read the result: Classify missing, type, enum, bound, pattern, and structural failures.
Next: Repair only bounded correctable representation.
Step 04 — Validate semantics/provenance
Why: Schema cannot know business truth.
Do: Check cross-field totals, ranges, references, dates, unique IDs, source citations, and requested completeness.
Read the result: Contradictory or unsupported values require re-extraction/escalation.
Next: Assign field-level confidence/missingness.
Step 05 — Run bounded repair
Why: Unlimited retries inflate cost and can drift facts.
Do: Send only schema errors plus original source context needed; cap attempts; require same schema hash and no unauthorized fact changes.
Read the result: Diff repaired output against raw and source.
Next: On final failure return structured reject.
Step 06 — Regression-test failure classes
Why: Intermittent formatting returns after model/schema changes.
Do: Test truncation, fence/prose, wrong type, unknown field, invalid enum, missing field, semantic contradiction, refusal, timeout, and oversized result.
Read the result: No invalid result crosses side-effect boundary.
Next: Monitor validation/repair rate by model/schema version.
Worked example
total says $9,800.Evidence collected
- JSON parse and schema both pass.
- Source invoice visually states $980.
- Line arithmetic equals $980.
- Repair prompt without source sometimes changes line items instead.
Decision: This is semantic inconsistency, not syntax. Blind repair can invent financial facts.
Actions taken
- Rejected output at invariant check.
- Re-extracted total with source region and line evidence.
- Required source locator and arithmetic check before acceptance.
Why this example matters: Typed output is not trustworthy until domain invariants and provenance pass.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Raw response/finish/schema/model metadata are preserved.
- Strict parse and schema validation pass.
- Semantic invariants and provenance pass.
- Repair attempts are bounded and diff-audited.
- Invalid output cannot trigger side effects.
- Failure metrics and regression fixtures cover known classes.
Rollback or safe recovery
- Disable candidate schema/model and restore prior pinned combination.
- Replay preserved raw outputs through prior validator for comparison.
- Quarantine affected downstream records and rebuild from source.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Same field fails repeatedly | Schema/provider capability or prompt ambiguity. | Simplify/split schema and test supported subset. |
| Output cuts off | Token/result size limit. | Paginate, shorten fields, or chunk task; do not close braces manually. |
| Repair raises costs | Retry loop treats systemic defect as random. | Stop after cap and fix schema/generation design. |
| Schema version drift | Producer and consumer use different contract. | Hash/version schema and reject unknown version. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Raw response metadata and schema hash.
- Syntax/schema/semantic/provenance validation report.
- Repair attempts and diffs.
- Accepted output or explicit reject with missing evidence.
- Regression and monitoring metrics.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points