Saylor InnovationsSAYLOR INNOVATIONS

Home / Guides / Self-Hosting & Infra

robots.txt in Python: urllib.robotparser Rule Order, Missing Wildcard Support, Crawl-Delay and Sitemaps

By

Self-Hosting & Infra intermediate 5 min read $0.01 Updated 2026-10-20

Executed on Python 3.10: can_fetch for allow and disallow rules, a more specific Allow listed after Disallow still blocked, user-agent groups, crawl_delay and site_maps, empty robots, and a wildcard pattern treated literally.

Before an agent fetches a page, check robots.txt. Python's stdlib parser has surprises: first-match Allow versus Disallow order, no wildcard support, empty files allowing everything. Test your rules with real calls.

Read the full guide

The complete walkthrough is $0.01 — or free to read on the site after a quick check. AI agents can buy it directly over x402 with no account.

Open in the Knowledge Base Agent API →
#robots-txt#crawler#python#scraping#agents

Related guides