SAYLOR The result you are building
Finished result: A deterministic pipeline that validates model output against a supported schema, distinguishes syntax/truncation/schema/semantic failures, retries with bounded repair, and never silently accepts corrupted or invented fields.
Use this guide when
- Model JSON is invalid, truncated, schema-incompatible, or intermittently missing fields.
- Downstream automation needs reliable typed records.
Do not use it as a substitute for
- Do not regex-patch arbitrary JSON into validity and call it correct.
- Do not coerce missing high-impact facts or drop validation errors silently.
Before you change anything
- Exact schema and provider/model structured-output capabilities.
- Raw response, finish/stop reason, token limits, and validation errors.
- Expected semantic invariants and acceptable null/reject behavior.
- Retry/cost/latency budget and side-effect boundary.
Understand the system before fixing it
Valid JSON is weaker than valid data. Syntax can pass while IDs, dates, totals, relationships, or evidence are wrong.
Truncation is not a schema-repair problem. Finish reason and incomplete tail require smaller output/pagination or more budget, not brace insertion.
Provider-supported schema subset matters. Unions, recursion, additional properties, defaults, and formats may behave differently; test the actual provider/model.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Parser error at end | Truncation/syntax | Finish reason and raw tail | Increase/split output only after confirming limit; never append guessed content. |
| Type/required error | Schema | JSON Pointer validation errors | Repair exact fields or redesign schema. |
| Schema passes; totals impossible | Semantic | Cross-field invariant/calculation | Reject and re-extract from source; schema alone insufficient. |
| Repair loop changes facts | Unsafe repair | Diff raw vs repaired values/provenance | Repairs may format only; factual change requires source evidence. |
Step-by-step procedure
01 Capture raw response metadata
Why: Repair destroys diagnosis if raw bytes and finish reason are lost. Do: Store sanitized raw output, model/version, request/schema hash, finish reason, usage, and timestamp. Read the result: Truncated/refused/tool-call outputs take different branches. Next: Do not log secrets/full sensitive prompts.
02 Parse without heuristic mutation
Why: Parser position identifies syntax class. Do: Attempt strict JSON parse once; record error offset/context. Read the result: If truncated, reduce/split/regenerate; if wrappers/fences, fix generation contract rather than global regex. Next: Never execute partial data.
03 Validate exact schema
Why: Loose validation lets unknown/mistyped data through. Do: Use pinned JSON Schema validator, reject unknown fields where intended, and return JSON Pointer errors. Read the result: Classify missing, type, enum, bound, pattern, and structural failures. Next: Repair only bounded correctable representation.
04 Validate semantics/provenance
Why: Schema cannot know business truth. Do: Check cross-field totals, ranges, references, dates, unique IDs, source citations, and requested completeness. Read the result: Contradictory or unsupported values require re-extraction/escalation. Next: Assign field-level confidence/missingness.
05 Run bounded repair
Why: Unlimited retries inflate cost and can drift facts. Do: Send only schema errors plus original source context needed; cap attempts; require same schema hash and no unauthorized fact changes. Read the result: Diff repaired output against raw and source. Next: On final failure return structured reject.
06 Regression-test failure classes
Why: Intermittent formatting returns after model/schema changes. Do: Test truncation, fence/prose, wrong type, unknown field, invalid enum, missing field, semantic contradiction, refusal, timeout, and oversized result. Read the result: No invalid result crosses side-effect boundary. Next: Monitor validation/repair rate by model/schema version.
Worked example
Starting problem: An invoice extractor returns valid JSON where line totals sum to $980 but total says $9,800.
Evidence collected
- JSON parse and schema both pass.
- Source invoice visually states $980.
- Line arithmetic equals $980.
- Repair prompt without source sometimes changes line items instead.
Decision: This is semantic inconsistency, not syntax. Blind repair can invent financial facts.
Actions taken
- Rejected output at invariant check.
- Re-extracted total with source region and line evidence.
- Required source locator and arithmetic check before acceptance.
Proof of completion: Accepted record matches source and arithmetic; raw/rejected attempt remains auditable; no downstream payment uses invalid total.
Why this example matters: Typed output is not trustworthy until domain invariants and provenance pass.
Verify, recover, and hand off
Completion tests
- Raw response/finish/schema/model metadata are preserved.
- Strict parse and schema validation pass.
- Semantic invariants and provenance pass.
- Repair attempts are bounded and diff-audited.
- Invalid output cannot trigger side effects.
- Failure metrics and regression fixtures cover known classes.
Rollback or safe recovery
- Disable candidate schema/model and restore prior pinned combination.
- Replay preserved raw outputs through prior validator for comparison.
- Quarantine affected downstream records and rebuild from source.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Same field fails repeatedly | Schema/provider capability or prompt ambiguity | Simplify/split schema and test supported subset. |
| Output cuts off | Token/result size limit | Paginate, shorten fields, or chunk task; do not close braces manually. |
| Repair raises costs | Retry loop treats systemic defect as random | Stop after cap and fix schema/generation design. |
| Schema version drift | Producer and consumer use different contract | Hash/version schema and reject unknown version. |
Reusable handoff record
- Raw response metadata and schema hash.
- Syntax/schema/semantic/provenance validation report.
- Repair attempts and diffs.
- Accepted output or explicit reject with missing evidence.
- Regression and monitoring metrics.
For agents
This guide also defines a structured diagnose/propose/execute contract for building an agent-facing tool around this workflow (a design reference, not a live endpoint on this site today):
Required inputs: context, evidence, constraints, success — same shape as the other guides in this series.
Returned output: diagnosis, plan, verification, handoff.
Refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: score confidence from the number and quality of independent observations, not from how familiar the error looks.
References
- https://json-schema.org/specification
- https://www.rfc-editor.org/rfc/rfc8259