SAYLOR Evaluate lexical, vector, hybrid, reranked, and filtered retrieval against task-specific relevance, freshness, coverage, latency, and leakage tests.
The result you're building
A versioned retrieval benchmark and release gate that measures whether the correct evidence is found, ranked, scoped, and cited for representative questions, including hard negatives and permission boundaries.
Use this guide when
- You are building RAG, semantic search, recommendations, or agent memory retrieval.
- Embedding model, chunking, index, filters, or reranker changes.
- Good-looking demonstrations do not predict production question coverage.
Do not use it as a substitute for
- Choosing a model from a public leaderboard without your own corpus and tasks.
- Evaluating only answer fluency instead of whether the supporting evidence was retrieved.
Before you change anything
- Collect these items first. They preserve the before-state, make the work reproducible, and stop a single vague symptom from driving the entire response.
- Versioned corpus snapshot, permission model, chunking, metadata, and index settings.
- Representative query set with relevance judgments and acceptable evidence.
- Hard negatives, no-answer cases, stale/conflicting documents, and tenant canaries.
- Candidate/retrieval/rerank traces with scores and filters.
- Recall, precision, ranking, citation, latency, cost, and leakage by slice.
Understand the system before fixing it
Provenance is part of the record
A value without source, observation time, transformation history, and known limitations cannot support a defensible automated decision.
Schema changes are product changes
Renames, units, null behavior, identifiers, and deleted fields can silently change decisions even when a pipeline still returns HTTP 200.
Retrieval and generation must be evaluated separately
A fluent model can hide missing evidence; a strong retriever can be undermined by generation. Preserve both traces.
The right metric follows the task
Top-1 accuracy, recall at k, nDCG, citation support, diversity, freshness, and latency matter differently for lookup, research, and discovery.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| Answer cites irrelevant chunk | Ranking/generation | Inspect candidates, rerank, and cited span | Relevant evidence was not retrieved/ranked or generation ignored it. |
| Known document never appears | Ingestion/chunking | Query exact terms and inspect index membership | Document is absent, filtered, over-chunked, or embedded incorrectly. |
| Private canary is retrieved | Authorization/filter | Run query under unrelated tenant identity | Security relies on post-retrieval prompt filtering. |
| Offline score rises but users fail | Benchmark drift | Compare production failures to evaluation slices | Test queries are unrepresentative or overfit. |
| Hybrid repeats near-duplicate chunks | Diversification | Measure unique sources and marginal relevance | Ranking maximizes similarity without coverage. |
Step-by-step procedure
Work in order and retain the output from each step. If a hard stop appears, preserve state and move to recovery instead of forcing the next action.
Step 01 — Define retrieval jobs and failure cost
Why: A precise boundary prevents a plausible fix from solving the wrong problem.
Do: Separate exact lookup, broad research, current policy, personal memory, recommendations, and no-answer detection. Set required source types and maximum leakage risk.
Read the result: Each query class has explicit relevance and safety criteria.
Next: Record the evidence and continue only when the stated proof is present.
Step 02 — Build a judged query set
Why: Symptoms are not enough; a baseline preserves the evidence needed to isolate the failing layer.
Do: Sample real tasks, include rare and adversarial cases, label relevant passages/sources, allowed filters, acceptable no-answer, and judgment confidence.
Read the result: Queries cover production slices without using evaluation answers for tuning.
Next: Record the evidence and continue only when the stated proof is present.
Step 03 — Snapshot the whole retrieval stack
Why: Inconsistent inputs create false differences and make later comparisons unreliable.
Do: Version source corpus, parser, chunker, embedding model, index, lexical analyzer, metadata, filters, reranker, and query rewrite.
Read the result: A benchmark run can be reproduced exactly.
Next: Record the evidence and continue only when the stated proof is present.
Step 04 — Trace every retrieval stage
Why: A decisive test reduces trial-and-error and limits unnecessary change.
Do: Record rewritten query, candidate IDs/scores, filter decisions, rerank, selected context, citation, latency, and cost.
Read the result: A failure can be assigned to ingestion, retrieval, ranking, filtering, or generation.
Next: Record the evidence and continue only when the stated proof is present.
Step 05 — Measure task and safety metrics
Why: The smallest reversible correction lowers the blast radius while preserving a recovery path.
Do: Calculate recall/precision/MRR/nDCG as appropriate plus source diversity, freshness, citation support, no-answer calibration, permission leakage, latency, and cost by slice.
Read the result: Release thresholds reflect actual consequence, not one aggregate score.
Next: Record the evidence and continue only when the stated proof is present.
Step 06 — Stress hard negatives and drift
Why: The happy path cannot expose replay, timeout, malformed-input, authority, or dependency failures.
Do: Add similar-but-wrong documents, stale versions, conflicting policies, duplicate chunks, multilingual queries, misspellings, and tenant canaries.
Read the result: The system rejects or qualifies misleading evidence instead of confidently retrieving it.
Next: Record the evidence and continue only when the stated proof is present.
Step 07 — Canary and monitor production
Why: A result is not complete until it remains observable and repeatable after the immediate fix.
Do: Shadow the new stack, compare disagreements, review sampled failures, and alert on empty retrieval, filter drift, source concentration, latency, and corpus age.
Read the result: Rollback is triggered automatically when required slices regress.
Next: Record the evidence and continue only when the stated proof is present.
Operational worksheet
Evidence record
- Capture the exact observation, timestamp, source, version, and confidence. Sanitize credentials and personal data before sharing the record.
- Versioned corpus snapshot, permission model, chunking, metadata, and index settings.
- Representative query set with relevance judgments and acceptable evidence.
- Hard negatives, no-answer cases, stale/conflicting documents, and tenant canaries.
- Candidate/retrieval/rerank traces with scores and filters.
- Recall, precision, ranking, citation, latency, cost, and leakage by slice.
Acceptance scoreboard
- Benchmark corpus, queries, judgments, models, chunking, index, and filters are versioned.
- Metrics match each retrieval job and report critical slices separately.
- No-answer, hard-negative, stale/conflict, multilingual, and typo cases are included.
- Tenant and permission canaries never cross boundaries.
- Failures can be localized to ingestion, retrieval, ranking, filtering, or generation.
- Production canary and rollback thresholds protect required recall, freshness, latency, and safety.
Minimum handoff record
- Versioned embedding and retrieval evaluation scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Worked example
Evidence collected
- Both documents contain similar headings.
- The older document has more inbound links and higher lexical score.
- Effective and superseded dates are metadata but not used in ranking.
- Evaluation lacks conflicting-version queries.
Decision: Freshness and authority are missing from the retrieval contract. Add version precedence and benchmark superseded-document cases.
Actions taken
- Marked effective and superseded relationships.
- Filtered or demoted inactive policy for current-policy queries.
- Added hard negatives using old versions.
- Required citations to show effective date.
Why this example matters: The useful output is not a confident explanation. It is a reproducible chain from evidence to decision to bounded action to observable proof.
Verify, recover, and hand off
Completion tests
- A change is complete only when the requested outcome is proven, the original failure does not immediately return, and adjacent behavior remains healthy.
- Benchmark corpus, queries, judgments, models, chunking, index, and filters are versioned.
- Metrics match each retrieval job and report critical slices separately.
- No-answer, hard-negative, stale/conflict, multilingual, and typo cases are included.
- Tenant and permission canaries never cross boundaries.
- Failures can be localized to ingestion, retrieval, ranking, filtering, or generation.
- Production canary and rollback thresholds protect required recall, freshness, latency, and safety.
Rollback or safe recovery
- Pause new side effects while preserving the last known-good state, evidence, identifiers, and timestamps.
- Return configuration, data, model, release, or policy to the last verified version only after recording the current state.
- Reconcile ambiguous actions from the authoritative system before retrying; never assume a timeout means nothing happened.
- Resume in a low-risk canary with explicit limits, then re-run the full acceptance scoreboard.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Answer cites irrelevant chunk | Relevant evidence was not retrieved/ranked or generation ignored it. | Inspect candidates, rerank, and cited span |
| Known document never appears | Document is absent, filtered, over-chunked, or embedded incorrectly. | Query exact terms and inspect index membership |
| Private canary is retrieved | Security relies on post-retrieval prompt filtering. | Run query under unrelated tenant identity |
| Offline score rises but users fail | Test queries are unrepresentative or overfit. | Compare production failures to evaluation slices |
Reusable handoff record
- Versioned embedding and retrieval evaluation scope, owner, exclusions, and success criteria.
- Sanitized evidence snapshot with source, time, version, and confidence.
- Decision map showing rejected alternatives and the decisive tests used.
- Ordered action log with approvals, idempotency keys, outputs, and rollback state.
- Acceptance results, remaining risks, review date, and escalation owner.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| target | object | Versioned environment, resource, identity, or workflow being evaluated. |
| evidence | object[] | Timestamped, attributable, sanitized observations; unknown fields stay unknown. |
| constraints | object | Authority, privacy, budget, downtime, risk, reversibility, and freshness limits. |
| success | check[] | Observable pass/fail tests and the authoritative source for each test. |
Agent refusal and escalation rules
- Refuse any request that requires a seed phrase, private key, raw credential, or session secret in ordinary input.
- Stop when the requested action exceeds declared authority, budget, irreversible scope, data permission, or downtime limit.
- Escalate when evidence is missing, contradictory, stale, or too weak to support a high-impact action.
- Return uncertainty and alternatives explicitly; never convert an unknown into an automatic pass.
Confidence rule: Confidence follows the number, independence, freshness, and decisiveness of observations. Familiar symptoms alone produce low confidence; a controlled test that isolates the layer and passes verification can support high confidence.
Official reference starting points