JavaScript Challenge

A JavaScript challenge is an interstitial page served in place of the content you asked for. There is no visible puzzle. The server holds the request and, in Cloudflare's own description, "intercepts the visitor from getting to the destination URL by holding the request and evaluating the browser environment for automated signals, and serving a challenge" (Cloudflare challenge pages documentation, read 2026-07-27). The page runs browser-only work: proof-of-work computation, environment probes, timing checks. Cloudflare says of its non-interactive form that "the visitor must wait until their browser finishes processing the JavaScript, which typically takes less than five seconds." Pass it and the server sets a clearance cookie and sends you on to the real content.

This breaks every plain HTTP client, and it breaks them quietly. A requests or httpx or Go client has nothing to execute, so it receives the interstitial, parses it as valid HTML, and stores it. No exception is raised. Your parser finds no products and writes an empty row, or worse, extracts the challenge page's boilerplate. A crawl that appears to be running at full speed can be harvesting interstitials for days.

Detection from the crawler's side is straightforward once you look for it. The interstitial is small, often a few kilobytes against a content page's hundreds. It may arrive with a 200 or a 403 and carry a meta refresh. A known cookie name appearing on retry is another tell; Cloudflare documents Turnstile Pre-clearance as issuing "a persistent clearance cookie (cf_clearance) upon successful human verification on an initial HTML page." The cheapest guard is a required-selector assertion per target with an alert when its miss rate crosses a threshold. The clearance cookie's lifetime is what makes solving worth doing at all, because one successful pass amortizes across every subsequent request in that session. We could not confirm a default duration from primary documentation, so measure it against your own targets rather than assuming.

Three responses are available. Run a real browser, pass the challenge once, and keep the cookie jar for the session. Buy a vendor that handles clearance for you, which is core to what Scrapfly, ZenRows, ScrapingBee and Scrapeless sell. Or drop the target, which is a legitimate answer when the content is not worth the per-session cost. Whichever you choose, treat cookie, exit IP and fingerprint as one unit: clearance is generally tied to the identity that earned it, so rotating the IP mid-session usually throws the cookie away and sends you back to the interstitial.