Crawl-Delay

Crawl-delay is a directive publishers place in robots.txt asking crawlers to wait a minimum number of seconds between requests, written as a line such as Crawl-delay: 10 inside a user-agent group. It is not part of the original robots exclusion standard and support is uneven. Google's robots.txt documentation, read on 2026-07-27, is explicit: "Google supports the following fields (other fields such as crawl-delay aren't supported): user-agent, allow, disallow, sitemap." Bing and Yandex have historically honored it, and most open-source crawlers read it. It applies alongside allow and disallow rather than instead of them, so a delay never grants access to a disallowed path.

Treat a published delay as the floor of a wider pacing policy rather than the whole of one. The rest is per-host concurrency held to one or two connections, adaptive pacing that slows down when origin response times rise, and honoring Retry-After on a 429 or 503 instead of retrying immediately. Frameworks expose this directly. Scrapy has DOWNLOAD_DELAY, CONCURRENT_REQUESTS_PER_DOMAIN, and an AutoThrottle extension that targets a latency rather than a fixed interval; Crawlee, Nutch, and node-crawler each carry an equivalent. Adaptive pacing usually beats a fixed number, because a delay that is comfortable at 3am is not comfortable during the target's traffic peak.

Politeness here is self-interested, not only courteous. Aggressive pacing is the fastest way to get an entire ASN or proxy subnet blocked, and that block outlives the job that caused it. It affects every crawl you run through the same pool, it is slow to lift, and replacing the burned addresses costs more than the throughput the burst bought. Sustained moderate rates finish more jobs than bursts that end in a block, and CDN-level defenses respond to request rate per origin long before they respond to anything about your fingerprint.

The decision is what per-host rate to set and how to justify it later. A workable default: respect a published Crawl-delay where one exists, and where none does, derive a delay from measured origin latency, for example one concurrent request per host with a delay near the observed median response time. Record that policy somewhere durable, along with the robots.txt fetch you based it on. When a compliance question or an abuse complaint arrives, the difference between a documented pacing policy and an undocumented one is the whole conversation.