Why Your Scraper Gets Blocked: Reading 403s, Challenges and Fingerprints

When a crawl starts returning 403s, the first purchase is almost always a residential proxy plan. Sometimes that works. More often the 403 came from a TLS handshake that looked like Python, the new exit addresses change nothing, and the team has bought a recurring bill instead of a fix.

The question worth asking first is which layer rejected you. There are six of them, they fail independently, and the fix for one has nothing to do with the fix for another. This guide is a diagnostic order of operations: capture the response, read it, identify the layer, then pick the class of tool that addresses that layer. It contains no evasion recipes and no walkthrough for defeating a challenge. Those decay within weeks of publication, and the paths worth recommending are the legitimate ones anyway.

If you are earlier than this and still choosing a category of tool, start with how to choose a web scraping tool and come back when something is actually failing.

Capture the whole response before you theorise

You cannot diagnose from a status code alone. Before doing anything else, log four things for a failing request and one known-good request to the same host:

1. The status code and the final URL after redirects.
2. Every response header, not just the ones your HTTP client surfaces by default.
3. Every Set-Cookie, including attributes.
4. The first few hundred bytes of the body, plus its total length.

Content length alone separates two very different failures. A 403 with an 8 KB HTML body is an interstitial. A 403 with a 200-byte body is a deny rule. They are different problems with different answers.

Log the request side too. If you are running a framework, make sure you know the exact header set and ordering it emits, not the set you configured. Crawlee does a fair amount of header and session management for you, which is helpful in production and confusing during diagnosis, because the request you reasoned about is not the request that went out.

The response triage table

This maps what you observe to the layer most likely responsible. It gives you a starting guess. Confirm it with the isolation tests in the next section before spending money.

The last row is the one teams reach latest and should reach earliest. If nothing you control changes the outcome, nothing you control is the cause.

Isolating the layer with three cheap tests

Each test holds everything constant except one variable. Run them in this order because they run from cheapest to most expensive.

Test one: change the IP, hold the client. Send the identical request from a different network. If a datacenter address fails and a residential one succeeds with byte-identical requests, you have an IP reputation problem and a proxy network is the right category. If both fail identically, stop looking at proxies. A residential proxy is expensive relative to a datacenter one, and buying it for a fingerprint problem is the single most common wasted purchase in this field.

Test two: change the client, hold the IP. Send the identical request from the same address using a browser-impersonating HTTP client, then a real browser. If the plain library fails and the impersonating client succeeds, the TLS or HTTP/2 fingerprint is the problem. If both HTTP clients fail and only the real browser succeeds, the target is checking the JavaScript environment, so you need rendering rather than a better handshake.

Test three: change the pace, hold both. Drop to one request every thirty seconds from a single session with cookies preserved. If failures disappear, you were rate limited or behaviourally scored, and no amount of infrastructure will substitute for pacing.

The three tests take an afternoon and tell you which of the layer sections below applies to you.

Layer one: IP reputation and geography

This is the layer everyone knows and the one that gets blamed for everything else. The edge assigns your source address a reputation from ASN, historical abuse, whether it belongs to a known hosting provider, and how many other clients are behind it right now. Datacenter ranges from major clouds carry poor reputation on consumer-facing sites for entirely rational reasons.

The signature is a short-bodied 403 or an immediate deny, consistent per exit address and inconsistent across a pool. Geo gating looks similar but sorts cleanly by country, and a 451 makes it explicit.

Bright Data is the reference vendor here, running residential, datacenter, ISP and mobile pools with scraping APIs layered over them. It is the correct answer when a target genuinely defeats everything cheaper and budget is not the binding constraint. It is the wrong answer for a two-person team scraping a few thousand pages a week: the account setup, minimum commitments and enterprise sales motion cost more attention than the problem is worth, and you will spend the first week in procurement rather than in code. If you are weighing it against its closest peer, the Bright Data vs Oxylabs comparison covers where each lands.

Two things worth knowing before you buy. A rotating proxy that changes address on every request will actively break any target that scores session continuity, because a session that teleports between three countries mid-checkout is more anomalous than one datacenter IP. And residential pools vary enormously in how their addresses are sourced, which is a diligence question as much as a technical one.

Layer two: TLS and HTTP/2 fingerprints

This layer is invisible from the application code, which is why it survives so long undiagnosed. Before a single HTTP header is parsed, the server has seen your TLS ClientHello: protocol version, cipher suite list and order, extensions, elliptic curves. That combination is stable per TLS library and differs measurably between Chrome, Firefox, Go's crypto/tls, Python's ssl module and everything else.

JA3, published by Salesforce in 2017, hashed those fields into a single value. JA4, which FoxIO released in September 2023, replaced it with a structured, human-readable family of fingerprints. Cloudflare's own engineering blog describes JA4 as more resistant to extension randomisation and covering additional protocols including QUIC, and the JA4+ suite extends the idea to HTTP with JA4H. The practical consequence is that the handshake happens before your User-Agent is read, so header spoofing cannot repair a mismatch beneath it.

HTTP/2 adds a second, independent surface. The client preface, the SETTINGS frame values and their order, the header compression behaviour and pseudo-header ordering all differ between real browsers and HTTP libraries. A client that presents a Chrome User-Agent over an HTTP/2 stack that does not look like Chrome is easier to classify than one that made no claim at all.

The signatures are distinctive: connection resets before any HTTP response, a plain library failing where an impersonating client succeeds from the same address, and failure that is completely insensitive to IP rotation.

There are two honest fixes. Use a client whose TLS and HTTP/2 behaviour genuinely matches the browser you are claiming to be, and keep the claim consistent across every layer. Or stop managing the transport yourself. ScrapeNinja is a useful shape here because it splits the two cases explicitly: a fast raw-request endpoint with Chrome fingerprint emulation, and a separate headless-browser endpoint for pages that need rendering. That split matches the diagnosis you just made. It is run by a single developer with no disclosed funding or third-party coverage, so a team that needs a contractual SLA and a support organisation should weigh that carefully against the larger vendors in the category.

Layer three: headless and browser environment signals

If your test three showed that only a real browser passes, the target is inspecting the JavaScript environment. Automation frameworks leave observable traces: property values that differ under automation, inconsistent screen and device metrics, font and codec availability that does not match the claimed platform, WebGL and canvas output that does not match the claimed GPU.

The signature is a 200 that contains an interstitial or an empty shell rather than content, or a challenge page that never resolves.

Playwright is the default open-source answer for anything that needs a real rendering engine. It drives Chromium, Firefox and WebKit through one API with auto-waiting and network interception, and Microsoft maintains it. It is the wrong tool for volume: a browser per request is expensive in memory and time, so for targets that return usable HTML without JavaScript you are paying an order of magnitude more per page for nothing. Diagnose before you reach for it.

Running browsers at scale is its own operational problem, which is what browser infrastructure vendors exist to absorb, with antidetect browsers sitting adjacent to it. Scrapeless bundles a cloud scraping browser with a universal scraping API and a proxy network under one account, which suits a team that would otherwise be running three contracts to answer one question. No funding round or substantial independent coverage is public for it, so a team betting a production pipeline on it should run its own reliability trial rather than lean on a track record that has not been established publicly.

Layer four: behavioural and session scoring

The fourth layer does not look at any single request. It looks at the shape of many: request rate, inter-request timing regularity, navigation order, whether cookies persist, whether you ever fetch the assets a browser would fetch, whether you arrive at deep URLs without ever touching an index.

The signature is failure that correlates with volume rather than identity: everything works for twenty minutes and then stops, a fresh session works immediately, and slowing down fixes it.

This is the layer where infrastructure spending helps least and design helps most. Persist cookies across a session. Vary intervals rather than sleeping a constant two seconds. Crawl in an order that a person could plausibly follow. Respect the concurrency the site can absorb, which is both the effective fix and the decent thing to do.

Crawlee is worth reaching for here specifically because request queuing, session pooling and retry behaviour are built in rather than bolted on, so the session model is something you configure instead of something you invent. Its Node implementation is considerably more mature than the newer Python port, so a Python-first team should weigh that before committing; the Scrapy vs Crawlee comparison covers the tradeoff. Note also that a crawling framework does nothing for layers one and two on its own. It organises your requests; it does not disguise them.

Layer five: account and geo gating

Some failures are not detection at all. Content behind a login is gated by an account, and scoring then attaches to that account rather than to your IP or fingerprint. The tell is that anonymous requests succeed and authenticated ones fail, which inverts the usual pattern and confuses teams who have only ever debugged the anonymous case.

Geo gating is simpler. Failure sorts cleanly by exit country, the body often says so, and a 451 states it outright. This is a targeting problem rather than a blocking one, and every serious proxy vendor sells country-level and often city-level selection.

Both of these should prompt a legal review before an engineering one. Circumventing an authentication boundary is a materially different act from fetching a public page, in terms of terms of service and, depending on jurisdiction, statute.

Layer six: policy, which you cannot engineer around

The sixth layer is the site, or its CDN, deciding that a whole class of crawler is not welcome regardless of how well it behaves. That is new in kind, and none of the tests above will find it.

Cloudflare published a post on 1 July 2026 setting out new AI traffic options, and it is worth reading in the original rather than in summary. Three points from it bear directly on collection work.

It splits automated traffic into three classifications and defines them by purpose rather than by identity: Search is "any behavior that collects or indexes your content, so it can answer questions about it later", Agent is "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now", and Training is "a crawler taking your content to train or fine-tune a model".

It then states that on 15 September 2026 it will set new defaults for each of those three. For all new domains onboarding to Cloudflare, Training and Agent will be blocked by default on the pages that display ads, while Search remains allowed by default. The post's stated reason is that those are the pages owners monetise and expect human attention on. Existing site owners who want something other than the defaults can set that in their Security settings at any point before 15 September, and Cloudflare says it will keep notifying customers as the date approaches.

The third point is the one most likely to surprise a crawler operator. Multi-purpose crawlers are, in Cloudflare's words, allowed or blocked "according to all of their behaviors", and the post names Googlebot, Applebot and BingBot as crawlers that will be blocked by a customer who has chosen to block Training. A crawler does not get partial credit for its well-behaved half.

What this does not tell you is how many sites will end up in each configuration, or how the classification behaves in edge cases. Those are empirical questions that will only be answerable after the date, so treat any confident projection of the impact, including one you read here, with suspicion.

Two consequences matter for anyone running collection at scale.

First, some share of your failures after mid-September will be policy rather than detection, and they will be indistinguishable from detection unless you look for the pattern. Policy blocks are uniform. They do not respond to IP rotation, fingerprint work, pacing or browser rendering, because nothing about your client is being scored. If a target fails identically across every variable you control, stop tuning and read the site's stated position.

Second, the identity direction is now the supported one. Cloudflare's documentation describes Web Bot Auth as an authentication method that uses cryptographic signatures in HTTP messages to verify that a request comes from an automated bot, and as a verification method for verified bots and agents. It builds on the HTTP Message Signatures specification, RFC 9421, plus two IETF drafts covering how a crawler publishes its public keys and attaches its identity to requests. In practice you host a key directory at a fixed well-known path, get verified, and then sign your requests. Note the word draft: parts of this are still moving, and Cloudflare's own implementation documents several RFC 9421 components it does not support.

That is the opposite posture to fingerprint evasion. You declare an identity and accept the access decision that follows, including when the answer is no. For anyone building an agent or a crawler with a durable name attached to it, that trade gets better over time rather than worse, because a signed identity is the only thing a site can grant an exception to.

Matching the fix class to the layer

The point of the diagnosis is that these do not substitute for each other.

When outsourcing the whole stack is the right call

If your diagnosis lands on layers one through three simultaneously, and the target is not core enough to your product to justify owning the problem, buying a managed web scraping API is a defensible engineering decision. You are paying a vendor to carry both the infrastructure and the compliance burden.

Scrapfly covers an unusually wide surface for its size: a single scraping endpoint, a cloud browser, an extraction API, a crawler API and an MCP server, with SDKs across five languages. That breadth is genuinely useful when your diagnosis is "several layers at once". It is bootstrapped and small, and its compliance claims are self-reported, so a regulated buyer that needs an audited vendor and a named support organisation should look at larger providers instead.

ZenRows takes the opposite shape: one universal endpoint that returns HTML or structured data, with a separate hosted scraping browser for teams that want to keep their own Playwright or Puppeteer logic. That second product matters, because the usual complaint about single-endpoint APIs is losing control of browser behaviour. It is still the wrong fit if your crawl logic is genuinely unusual, since the core product's value is precisely that it hides the decisions you would otherwise be making.

Scrape.do is worth a look when your targets are a handful of well-known sites, because its prebuilt endpoints return structured JSON for common targets without you maintaining parsers, and the vendor bills per successful request. It is closed-source and cloud-only; a self-hosted browser has been listed as upcoming but has not shipped, so any team with an on-premises requirement should rule it out now rather than after procurement. Treat its advertised pool size, like every vendor's, as a marketing figure until you have measured your own success rate against your own targets.

Note the pattern in all three: pay-per-success billing and freemium tiers exist because failure rates against hard targets are real and vendors know it. Pricing shapes in this category change often, so check current terms with the vendor rather than trusting any figure quoted in a guide, including this one.

What this guide will not do

There is no CAPTCHA-solving walkthrough here and no recipe for defeating a specific challenge version. Two reasons, one practical and one not.

The practical one: those pages are obsolete on a timescale of weeks. Anti-bot vendors ship new challenge versions continuously, and a guide organised around defeating version N is worthless against version N plus one. A guide organised around diagnosis stays true, because the layers themselves change slowly even as the implementations churn.

The other reason is that the recommended paths are the legitimate ones. Identify your crawler and accept the answer. License the data from a source that has the right to sell it. Buy a vendor that has taken on the compliance obligation and can tell you what it does and does not do. Those are the options that will still work in eighteen months, which is more than a challenge bypass can claim.

Where to go from here

Run the three isolation tests before you buy anything. Most teams discover they were debugging the wrong layer, and the correction is usually cheaper than the purchase they were about to make.

If the diagnosis points at infrastructure, the buying guide linked at the top walks through the three categories in more detail, and the 2026 stack overview covers how teams are currently assembling them. If you want the underlying concept rather than the tooling, the anti-bot system entry is the shortest useful definition.

And if the answer turns out to be layer six, there is nothing left to build. Read the site's stated position and decide whether to identify your crawler, license the data, or leave the target alone.