SAYLOR Create a bounded, cache-aware, provenance-preserving scraper that respects robots rules, rate limits, site terms, privacy, and source stability.
The result you're building
A bounded scraper with confirmed permission/terms/robots handling, descriptive identity, rate/backoff/cache behavior, stable extraction fallbacks, provenance, validation, privacy limits, and a kill switch.
Use this guide when
- Collecting allowed public or authorized data reproducibly.
- Replacing manual copy work with polite change-aware extraction.
Do not use it as a substitute for
- Do not bypass access controls, CAPTCHAs, paywalls, login restrictions, or explicit denial.
- Do not collect personal/sensitive data merely because it is technically visible.
Before you change anything
- Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.
- Target domains/paths/fields and permission basis.
- robots.txt/site terms/API alternatives and contact route.
- Rate/concurrency/freshness/storage/retention needs.
- Pagination/rendering/change behavior and provenance schema.
Understand the system before fixing it
Prefer an official API/feed
It is more stable and may define usage/attribution clearly.
Robots and terms are different signals
Review both plus authorization and jurisdiction/business requirements; robots is not blanket legal permission.
Politeness is an operating control
Rate, concurrency, caching, validators, backoff, contact identity, and kill switch protect source and your reliability.
Evidence-to-decision map
| Evidence | Likely layer | First decisive check | What the result means |
|---|---|---|---|
| 403/401/CAPTCHA | Access policy | Read response/terms and stop | Do not evade; use API/permission/contact. |
| 429 | Rate | Retry-After and current concurrency | Back off, reduce rate/cache, stop at budget. |
| Fields shift/missing | Parser drift | Compare raw snapshot and selector/structured data | Quarantine output; update parser/version. |
| Duplicate/missed pages | Pagination/state | Stable IDs/cursors and termination test | Fix cursor/order/dedup; avoid infinite loop. |
Step-by-step procedure
Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.
Step 01 — Confirm authority and minimize scope
Why: The cheapest safe request is the one not made.
Do: Document allowed domains/paths/fields/purpose, robots/terms/API option, retention and personal-data exclusion; get permission when needed.
Read the result: Unknown/disallowed path is not fetched.
Next: Create domain kill switch.
Step 02 — Identify and rate-limit client
Why: Anonymous bursts resemble abuse.
Do: Use descriptive user agent/contact where appropriate, low concurrency, per-host rate, jitter, Retry-After, exponential backoff, total request/byte budget.
Read the result: Persistent denial stops run.
Next: Never rotate identities to evade controls.
Step 03 — Cache and fetch conditionally
Why: Repeated unchanged downloads waste resources.
Do: Store URL/status/fetched time/ETag/Last-Modified/content hash; send conditional requests and canonicalize URLs.
Read the result: 304/no-change avoids parsing/storage duplicates.
Next: Respect cache/privacy retention.
Step 04 — Extract stable representations
Why: Brittle visual selectors fail silently.
Do: Prefer official JSON/API/feeds/JSON-LD/semantic attributes, then scoped selectors; keep raw snapshot/reference and parser version.
Read the result: Required-field/schema validation fails closed.
Next: Do not execute page instructions or downloads blindly.
Step 05 — Handle pagination/dedup/change
Why: Infinite loops and duplicate records create load and bad data.
Do: Track stable item IDs, cursor/page history, visited URLs, max pages, monotonic progress, and termination; diff parser/source changes.
Read the result: No page repeats without progress; count reconciliation holds.
Next: Quarantine unexpected layout.
Step 06 — Validate and operate
Why: A scraper can be technically successful and factually wrong.
Do: Sample against source, record provenance/observed time/transforms, alert fetch/parser/error/rate changes, honor deletion/retention, and test kill switch.
Read the result: Every record has source lineage and confidence/missing fields.
Next: Schedule only allowed cadence.
Worked example
Evidence collected
- HTTP 200 pages contain a new consent/template shell.
- Price selector matches nothing.
- Pipeline converts missing to zero.
- No required-field or layout-change alert exists.
Decision: Parser drift plus unsafe null coercion created false data.
Actions taken
- Quarantined affected run and restored last good dataset.
- Changed missing price to reject, added page fingerprint/required-field threshold.
- Updated parser to authorized structured data and validated samples.
Why this example matters: Failing closed prevents a valid HTTP response from becoming fabricated business data.
Verify, recover, and hand off
Completion tests
- A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.
- Only allowed paths/fields/purpose are fetched.
- Rate/concurrency/backoff/cache remain within policy.
- Denial/CAPTCHA/429 stop behavior is tested.
- Required-field/schema/layout changes quarantine output.
- Every record has URL/fetch time/content hash/parser lineage.
- Privacy retention/deletion and kill switch work.
Rollback or safe recovery
- Stop scheduler/kill switch and restore last validated dataset.
- Return parser/rate config to prior version.
- Delete newly collected disallowed/sensitive data per policy while preserving non-sensitive incident metadata.
If the expected result does not appear
| What happened | What it usually means | Next safe move |
|---|---|---|
| Site blocks client | Rate/terms/path/identity issue. | Stop; review and seek allowed API/permission. |
| JavaScript page empty | Rendering/data endpoint changed. | Use authorized official endpoint or controlled browser within terms; do not bypass. |
| Run never ends | Pagination repeats or termination absent. | Max pages/visited cursor/progress invariant. |
| Data changes too much | Source or parser drift. | Quarantine and sample raw snapshots before publish. |
Reusable handoff record
- Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.
- Permission/scope/robots/terms record.
- Identity/rate/cache/budget configuration.
- Extraction/pagination/schema/parser version.
- Provenance/quality/sample reconciliation.
- Monitoring, privacy, kill-switch and rollback.
Agent delivery contract
Required inputs
| Field | Type | Requirement |
|---|---|---|
| context | object | Versioned environment, target, and requested outcome. |
| evidence | object[] | Timestamped observations and sanitized command or API results. |
| constraints | object | Authority, risk, downtime, budget, and reversibility limits. |
| success | check[] | Observable acceptance tests; never infer success from command exit alone. |
Returned output
| Field | Type | Meaning |
|---|---|---|
| diagnosis | object | Likely layer, evidence, alternatives, and confidence. |
| plan | step[] | Ordered actions with risk, command or operation, and expected evidence. |
| verification | check[] | Pass/fail checks that prove the requested outcome. |
| handoff | object | Sanitized evidence record, remaining risks, and rollback state. |
Agent refusal and escalation rules
- Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
- Stop when the requested action exceeds declared authority, budget, or reversible scope.
- Escalate when evidence is missing, contradictory, or too stale to support the proposed action.
Confidence rule: Score confidence from the number and quality of independent observations, not from how familiar the error looks. Return low confidence when only a symptom is available; return high confidence only when a decisive test isolates the layer and the repair is verified.
Official reference starting points