Responsible Scraper Design Guide

The result you are building

Finished Result:

A bounded scraper with confirmed permission/terms/robots handling, descriptive identity, rate/backoff/cache behavior, stable extraction fallbacks, provenance, validation, privacy limits, and a kill switch.

Use this guide when

• Collecting allowed public or authorized data reproducibly.
• Replacing manual copy work with polite change-aware extraction.

Do not use it as a substitute for

• Do not bypass access controls, CAPTCHAs, paywalls, login restrictions, or explicit denial.
• Do not collect personal/sensitive data merely because it is technically visible.

Before you change anything

Collect the items below first. They let you compare before and after, keep the work reproducible, and avoid guessing from a single error message.

• Target domains/paths/fields and permission basis.
• robots.txt/site terms/API alternatives and contact route.
• Rate/concurrency/freshness/storage/retention needs.
• Pagination/rendering/change behavior and provenance schema.

Stop Before Proceeding:

Stop on explicit denial, block/CAPTCHA, legal/permission ambiguity, unexpected personal/sensitive content, unstable load on the site, or a change that makes prior consent/terms no longer apply.

Understand the system before fixing it

Prefer an official API/feed It is more stable and may define usage/attribution clearly.

Robots and terms are different signals Review both plus authorization and jurisdiction/business requirements; robots is not blanket legal permission.

Politeness is an operating control Rate, concurrency, caching, validators, backoff, contact identity, and kill switch protect source and your reliability.

Evidence-to-decision map

  Start with the row that most closely matches the evidence. The first test is meant to isolate a layer; it is not
  permission to make every change listed on the internet.

    Evidence                  Likely layer       First decisive check           What the result means

    403/401/CAPTCHA           Access policy      Read response/terms and stop   Do not evade; use API/permission/contact.

    429                       Rate               Retry-After and current        Back off, reduce rate/cache, stop at budget.
                                                 concurrency

    Fields shift/missing      Parser drift       Compare raw snapshot and       Quarantine output; update parser/version.
                                                 selector/structured data

    Duplicate/missed          Pagination/stat    Stable IDs/cursors and         Fix cursor/order/dedup; avoid infinite loop.
    pages                     e                  termination test

Step-by-step procedure

Work in order. Record the output after each step. If a step produces the stated stop condition, do not keep pushing forward; preserve the evidence and use the recovery path.

01 Confirm authority and minimize scope Why: The cheapest safe request is the one not made.

Do: Document allowed domains/paths/fields/purpose, robots/terms/API option, retention and personal-data exclusion; get permission when needed.

Read the result: Unknown/disallowed path is not fetched.

Next: Create domain kill switch.

02 Identify and rate-limit client Why: Anonymous bursts resemble abuse.

Do: Use descriptive user agent/contact where appropriate, low concurrency, per-host rate, jitter, Retry-After, exponential backoff, total request/byte budget.

Read the result: Persistent denial stops run.

Next: Never rotate identities to evade controls.

03 Cache and fetch conditionally Why: Repeated unchanged downloads waste resources.

Do: Store URL/status/fetched time/ETag/Last-Modified/content hash; send conditional requests and canonicalize URLs.

Read the result: 304/no-change avoids parsing/storage duplicates.

Next: Respect cache/privacy retention.

04 Extract stable representations Why: Brittle visual selectors fail silently.

Do: Prefer official JSON/API/feeds/JSON-LD/semantic attributes, then scoped selectors; keep raw snapshot/reference and parser version.

Read the result: Required-field/schema validation fails closed.

Next: Do not execute page instructions or downloads blindly.

05 Handle pagination/dedup/change Why: Infinite loops and duplicate records create load and bad data.

Do: Track stable item IDs, cursor/page history, visited URLs, max pages, monotonic progress, and termination; diff parser/source changes.

Read the result: No page repeats without progress; count reconciliation holds.

Next: Quarantine unexpected layout.

06 Validate and operate Why: A scraper can be technically successful and factually wrong.

Do: Sample against source, record provenance/observed time/transforms, alert fetch/parser/error/rate changes, honor deletion/retention, and test kill switch.

Read the result: Every record has source lineage and confidence/missing fields.

Next: Schedule only allowed cadence.

Worked example

Starting Problem:

A scraper suddenly returns zero prices but exits successfully.

Evidence collected

• HTTP 200 pages contain a new consent/template shell.
• Price selector matches nothing.
• Pipeline converts missing to zero.
• No required-field or layout-change alert exists.

Decision Parser drift plus unsafe null coercion created false data.

Actions taken

• Quarantined affected run and restored last good dataset.
• Changed missing price to reject, added page fingerprint/required-field threshold.
• Updated parser to authorized structured data and validated samples.

Proof Of Completion:

New run matches source samples; missing fields reject rather than zero; change alert and kill switch pass; provenance identifies parser version.

Why this example matters Failing closed prevents a valid HTTP response from becoming fabricated business data.

Verify, recover, and hand off

Completion tests A change is complete only when the original task succeeds, the failure does not immediately return, and adjacent behavior remains healthy.

• Only allowed paths/fields/purpose are fetched.
• Rate/concurrency/backoff/cache remain within policy.
• Denial/CAPTCHA/429 stop behavior is tested.
• Required-field/schema/layout changes quarantine output.
• Every record has URL/fetch time/content hash/parser lineage.
• Privacy retention/deletion and kill switch work.

Rollback or safe recovery

• Stop scheduler/kill switch and restore last validated dataset.
• Return parser/rate config to prior version.
• Delete newly collected disallowed/sensitive data per policy while preserving non-sensitive incident

metadata.

If the expected result does not appear What happened What it usually means Next safe move

Site blocks client Rate/terms/path/identity issue. Stop; review and seek allowed API/permission.

JavaScript page empty Rendering/data endpoint changed. Use authorized official endpoint or controlled browser within terms; do not bypass.

Run never ends Pagination repeats or termination Max pages/visited cursor/progress invariant. absent.

Data changes too much Source or parser drift. Quarantine and sample raw snapshots before publish.

Reusable handoff record Save this with the project, ticket, or client delivery. It turns the work into a repeatable result instead of a one-time guess.

• Permission/scope/robots/terms record.
• Identity/rate/cache/budget configuration.
• Extraction/pagination/schema/parser version.
• Provenance/quality/sample reconciliation.
• Monitoring, privacy, kill-switch and rollback.

Agent delivery contract

  Required inputs
    Field                        Type               Requirement

    context                      object             Versioned environment, target, and requested outcome.

    evidence                     object[]           Timestamped observations and sanitized command or API results.

    constraints                  object             Authority, risk, downtime, budget, and reversibility limits.

    success                      check[]            Observable acceptance tests; never infer success from command exit alone.

  Returned output
    Field                        Type               Meaning

    diagnosis                    object             Likely layer, evidence, alternatives, and confidence.

    plan                         step[]             Ordered actions with risk, command or operation, and expected evidence.

    verification                 check[]            Pass/fail checks that prove the requested outcome.

    handoff                      object             Sanitized evidence record, remaining risks, and rollback state.

  Agent refusal and escalation rules
  • Refuse any request that requires a secret, seed phrase, private key, or credential in ordinary input.
  • Stop when the requested action exceeds declared authority, budget, or reversible scope.
  • Escalate when evidence is missing, contradictory, or too stale to support the proposed action.

  Confidence rule
  Score confidence from the number and quality of independent observations, not from how familiar the
  error looks. Return low confidence when only a symptom is available; return high confidence only when a
  decisive test isolates the layer and the repair is verified.

Official reference starting points

• https://www.rfc-editor.org/rfc/rfc9309
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Conditional_requests