Exponential Backoff

Exponential backoff is the standard retry policy for transient failures: wait a short interval after the first failure, then multiply the wait after each subsequent failure, up to a ceiling. The usual form is delay equals base times multiplier raised to the attempt number, bounded by a cap. A base of 1 second with a multiplier of 2 and a cap of 60 seconds gives 1, 2, 4, 8, 16, 32, 60. Jitter is the part people leave out and the part that matters at fleet scale. Without randomisation, every worker that failed during the same outage retries at the same instant, and the recovery attempt becomes a second load spike. Full jitter replaces the computed delay with a uniform random value between zero and that delay, which spreads the retries across the whole window.

Retry the failures that a later attempt can plausibly fix. HTTP 429 qualifies: MDN describes it as indicating "the client has sent too many requests in a given amount of time" and notes that "a Retry-After header may be included to this response to indicate how long a client should wait before making the request again." When that header is present, honour it and let it override your computed delay, since it is the only number in the exchange that reflects the server's actual state. Retry 503, connection resets, read timeouts and TLS handshake failures. Do not retry 404, do not retry a malformed request, and do not retry a 403 that represents a hard block, because the block is a decision about your identity rather than a transient condition and repeating the call only confirms the pattern that produced it.

Cap the total spend. A retry budget expresses retries as a fraction of total requests over a rolling window, commonly around 10 percent, and stops retrying entirely once the fraction is exceeded. This is what prevents a target's bad hour from becoming your bandwidth bill. Under a full outage, an unbudgeted policy with three retries triples your billed volume for zero returned data, and on per-GB proxy pricing the failed responses are usually still metered. Pair the budget with a circuit breaker per target host so a sustained failure rate parks that host for a few minutes instead of grinding against it.

The policy worth writing down has four parts: which errors are retryable, the base, multiplier, cap and jitter form, a Retry-After override, and a budget that terminates the whole thing. Frameworks give you most of this already. Scrapy has retry middleware with a configurable retry count and status list, and Crawlee retries failed requests and sets persistently failing ones aside. Neither default knows your billing model, so the budget is still yours to set.