Web Application Firewall (WAF)

A web application firewall is the rule-based filtering layer that sits in front of an origin server and decides which requests reach it. Cloudflare's own documentation puts it plainly: the WAF "checks incoming web and API requests and filters undesired traffic based on sets of rules called rulesets" (Cloudflare WAF docs, read 2026-07-27). It is worth separating this from the bot-management product that usually ships alongside it. On that same page Cloudflare lists its bot solutions under "Related products," not as a WAF component. Scraping teams routinely blame the WAF for blocks that came from bot scoring, and blame bot scoring for blocks that came from a static rule, which sends the debugging effort in the wrong direction.

The WAF has its own trip wires that crawlers hit by accident, and none of them involve a bot score. Cloudflare's WAF is built from managed rules, custom rules and rate limiting rules. Managed rulesets match known-bad request shapes, and an odd header combination or a stray SQL-looking query string can match one even when the traffic is harmless. Custom rules cover the blunt instruments: country blocks, ASN blocks that catch entire hosting providers, and path rules that treat anything under /api or /search as sensitive. Rate limiting rules count requests per identifier per window and fire deterministically. A rule-based block is reproducible, which is the useful diagnostic property: the same request blocked ten times out of ten is a rule, while a request that succeeds six times out of ten is a score.

The vendors a scraping team meets are Cloudflare, Akamai, AWS WAF, Imperva, DataDome, Fastly and F5, and each responds to a block in a recognisable way. Cloudflare surfaces numbered 1xxx errors in the response body, of which error 1020 is documented as "A client or browser is blocked by a Cloudflare customer's Firewall Rules (deprecated)"; the docs page does not state which HTTP status accompanies it, so treat the commonly cited 403 as an observation from the field rather than a documented guarantee. Akamai returns a reference-number page, DataDome returns a distinctive interstitial. Response headers, cookie names and the body's exact shape are how you identify the stack.

The decision this informs is to fingerprint the defense before choosing the countermeasure, because the fixes have nothing in common. A geo or ASN rule is answered by changing exit network, which is what brightdata and oxylabs sell. A rate rule is answered by pacing and identifier spread. A managed-ruleset match is answered by cleaning up your request, not by buying better proxies. Only a behavioral score justifies the cost of full browser rendering, which is what managed APIs such as zenrows and scrapfly bundle in.