How to Read a Web Scraping Benchmark: Success Rate, Sample Bias, Sponsorship

In the table Zyte reprinted from Proxyway's Web Scraping API Report 2025, ZenRows appears twice: 70.39% success under one request-rate condition and 31.76% under another. Same provider, same report, a gap of nearly 39 points. Which of those two numbers ends up in a comparison article depends on who is doing the quoting.

A team with three vendors to evaluate and two weeks to do it will read a published report, take the top names, and start a trial. That is a reasonable use of someone else's work. The trouble is that reports differ in what they measure, and the summary figures that circulate afterwards in vendor blog posts and comparison articles rarely carry the methodology that produced them.

We publish no benchmarks of our own. Every number below is attributed to whoever published it, with a date. What follows is a reading method: the questions that separate a benchmark you can act on from a number that means nothing outside the run that produced it.

Every publisher defines success differently

The most consequential choice a benchmark publisher makes is what counts as a successful request, and there is no standard. Four definitions in circulation:

Move from the first row to the last on identical traffic and the reported figure falls, sometimes by tens of points. A vendor quoting 99% and an independent tester quoting 68% may both be honest and may both be describing the same service on the same day.

Proxyway's Web Scraping API Report 2025 is a good example of a publisher that answers the question. Read on 27 July 2026, it states that it tested "11 major providers of web scraping APIs" against 15 targets spanning e-commerce, search engines and social media, and that "we ran the bulk of our tests in October 2025." On what counted as a success, it says results were validated by examining the response code, page size and page title, with CSS selectors used in specific cases. That is row two edging toward row three in the table above: stronger than a bare 200, weaker than a validated record. Read the method and you at least know which row you are in.

This is also why the reprint chain matters. Zyte, which the report placed first, published its own summary and reprinted a table showing its API at 93.14% and 85.89% under two different conditions, with other participants below that. Separately, a NewsCatcher blog post citing the same report tells readers that "even the most reliable web scraping API services average 85-98% success rates." That upper bound does not appear in the reprinted table. Nobody lied. A range got rounded outward as it passed through a third hand, and the version most people will read is the loose one.

When you see a success rate with no definition attached, treat it as a marketing figure until proven otherwise. When you see one with a definition, write the definition down next to the number, because you will need it when you compare against the next report.

Target selection sets the ranking before the first request

The site list does more to fix the outcome than anything else in the method. Choose 15 of the most heavily defended commerce and social properties on the web and you have built a test of anti-bot bypass investment. Choose 15 ordinary content sites and you have built a test of latency and price. The same eleven vendors will rank differently in each, and neither ranking is wrong.

The distortion shows up in the aggregates. Writing about the Proxyway 2025 results, Bright Data's blog notes that Shein averaged a 21.88% success rate across the providers tested. One target like that, in a ten or fifteen site sample, drags every average toward it and compresses the differences between vendors on the other fourteen. If Shein is not on your target list, that portion of the score is noise for your purposes. If Shein is your target list, the aggregate is the wrong number entirely and you want the per-site breakdown.

So the first question to ask of any scraping benchmark is what it tested and whether that overlaps with what you scrape. Who won comes second. A report covering web scraping APIs against protected retail says little about scraping government open-data portals, and a proxy network evaluation on a search engine says little about long-tail forums. If your targets are absent from the sample, the report tells you about vendor engineering priorities, which is real information, but not about your success rate.

Related reading on why the underlying variance exists at all: why scrapers get blocked and the anti-bot system definition.

A single run on a single date is a snapshot

Anti-bot vendors ship rule changes continuously. A benchmark run over a few weeks in October describes October. It does not support a claim about reliability, which is a property of a system over time and can only be measured over time.

One-off testing is expensive and worth doing. The problem is the sentence that gets built on top of it. "Provider X is the most reliable scraping API" is a much stronger claim than "Provider X returned the most successful responses across 15 sites in October 2025," and only the second one was measured. Any report that runs once per year is describing a moving target with a still photograph.

Two practical consequences. First, prefer reports that publish their test window, because a report that omits its dates is omitting the part of itself that expires first. Second, when a vendor's marketing cites a benchmark from more than a couple of quarters back, treat the citation as evidence of past investment rather than present performance. The useful version of a reliability claim is a time series, and almost nobody publishes one.

Latency numbers are about request shape as much as vendor speed

Response time figures diverge by an order of magnitude between publishers, usually for structural reasons. Consider what sits inside a reported "average response time":

• Whether requests were issued sequentially or concurrently, and at what rate.
• Whether retries and challenge-solving time count against the clock or are excluded.
• Whether the measurement stops at first byte, at the complete HTML, or at a parsed record.
• Whether JavaScript rendering was enabled, which can be the difference between one second and thirty.
• Whether the vendor's pool was warm from prior traffic or cold.

Which is what the ZenRows pair at the top of this guide is really showing. In that same reprinted table, several providers carry two success figures under two request-rate conditions, and the gap between the pair says more than either figure alone. Zyte's own pair moves from 93.14% to 85.89%, about seven points. ZenRows drops nearly 39. A vendor that holds its rate under pressure and one that collapses can look similar at the top line and behave nothing alike in production. If a report gives you only one success figure, you cannot tell which kind you are buying.

The same logic applies to cost. Per-request pricing, credit multipliers for rendering, and retry billing interact with success rate in ways no single number captures. We cover the mechanics separately in scraping API credit pricing.

Who paid, and does the report say so

Independent testing in this niche is expensive, and the money has to come from somewhere. Nearly every publisher has commercial relationships with the vendors it ranks. What varies is whether it names them.

Two examples worth studying, both checked in July 2026:

AIMultiple publishes a funding page that is blunter than most. Read on 27 July 2026, it says: "Vendors either purchase consulting projects or request benchmarks. Only benchmark results are publicly shared and therefore, they present the most important conflict of interest." It commits to flagging the relationship at the point of use: "If an AIMultiple customer's product or service is mentioned on any work of AIMultiple, AIMultiple will notify the readers about this."

The commitment holds up on the page itself. Its web scraping benchmark, read the same day, carries the line "All providers in this benchmark are AIMultiple customers. Our team ensures objectivity," and adds that all providers contributed by providing part or all of the credits used. That page reports an analysis of 30 million web requests and places Bright Data first at 89% API coverage and 98% unblocking rates, with Zyte named as the lowest-priced unblocker at around two seconds. Those are AIMultiple's figures, not ours, and we have not reproduced them.

Sit the disclosure next to the ranking and you have the full picture: every vendor in the sample is a paying customer, and every vendor donated the credits the test ran on. That does not make the measurements wrong. It tells you the sample was assembled through a commercial relationship, so the ranking answers "who among AIMultiple's customers performed best" rather than "who is best." A vendor that declined to participate cannot lose, and cannot appear.

Proxyway takes a different shape. Its Web Scraping API Report 2025, read on 27 July 2026, carries an affiliate notice at the top: "We use affiliate links. They let us sustain ourselves at no cost to you." On method it is specific, and it describes a real safeguard: "We informed the participants about our general methodology but didn't disclose the list of websites in advance," with the stated intent of avoiding preemptive optimization. Withholding the target list is the most effective guard against a benchmark being gamed, and it deserves credit.

What the report does not contain is any statement about whether the tested providers paid toward it. That absence is not an accusation. It is just why affiliate revenue is worth keeping in mind while you read the ranking: affiliate programs are not uniform across vendors, and the ones that pay best are not chosen at random.

Neither of these is a scandal. Both are the visible, documented version of an arrangement that exists everywhere in this space. The reports to distrust are the ones where you cannot find any funding statement at all, or where the disclosure is a single line in a footer while the top-ranked vendor's logo sits above the fold.

A rough hierarchy of evidence quality, worst to best:

What an open harness looks like

The benchmarks category on this site is mostly academic and open-source web agent evaluation, not vendor scraping tests. That is deliberate, and those projects are the clearest worked examples of methodology you can actually inspect. They evaluate agents rather than proxies, but the disciplines transfer directly.

WebArena is the reference case. It self-hosts fully functional websites across four domains the paper names as e-commerce, social forum discussions, collaborative software development and content management, supported by wiki and map utilities, and instantiates 812 tasks from 241 intent templates. Scoring is functional correctness: programmatic reward functions check the resulting environment state rather than matching an action sequence. The original paper, published on arXiv in 2023 by researchers including Shuyan Zhou and Frank F. Xu with Carnegie Mellon affiliation, reports a best GPT-4 agent at 14.41% end-to-end task success against a human baseline of 78.24%. Both numbers are in the abstract, the environment is reproducible because it is sandboxed, and anyone disputing a score can rerun it. Rerunnability is the argument for open harnesses.

Reproducibility costs something, though. WebArena's sites are simulated, so a score there does not tell you how an agent handles live anti-bot defenses, session expiry, or a checkout flow that changed last Tuesday. If your question is whether something survives contact with production, its determinism is exactly what makes it the wrong test.

Mind2Web, from the Ohio State University NLP Group and published at NeurIPS 2023, takes the opposite position: 2,350 tasks across 137 real websites and 31 domains, averaging 7.3 actions per task, with explicit cross-task, cross-website and cross-domain generalization splits. The generalization splits are the design idea worth stealing. A benchmark that tests only on sites the agent has seen measures memorization. Mind2Web is the wrong choice if you need live interaction, since the dataset is built from captured DOM snapshots, screenshots and action traces rather than live sessions.

WebCanvas exists to close that gap. It is MIT-licensed, runs in live online environments, and scores with a key nodes methodology that breaks a task into phases instead of one pass/fail check. Its Mind2Web-Live dataset publishes 542 tasks with 2,439 intermediate evaluation states. Partial-credit scoring is the honest response to a real problem: a binary success rate treats an agent that got nine steps right and failed the tenth identically to one that failed immediately. The cost is that live environments drift, so a WebCanvas score is dated by construction. It is the wrong benchmark for anyone who needs two runs six months apart to be strictly comparable.

OSWorld widens the frame to full computer use across Ubuntu, Windows and macOS: 369 real computer tasks, each with a custom execution-based evaluation script, released under CC BY-SA 4.0 by researchers from the University of Hong Kong, Salesforce Research, Carnegie Mellon and Waterloo. The abstract reports that humans accomplish over 72.36% of the tasks while the best model reaches 12.24%. Execution-based checkers are the strongest form of scoring available, because they inspect the resulting state rather than the agent's account of it. OSWorld is the wrong benchmark if your work is browser-only, since much of the task set covers desktop applications you will never automate.

GAIA, built by researchers at Meta's FAIR team and Hugging Face and posted to arXiv in November 2023, devises 466 questions in three difficulty levels, hosted with a public leaderboard Space. Many questions require live browsing or search to answer. The abstract reports human respondents at 92% against 15% for GPT-4 equipped with plugins, which is the widest human-model gap of any benchmark named here. Its usefulness is also its exposure: a public leaderboard on a fixed question set invites overfitting, and any leaderboard-derived number should be read with that in mind. GAIA is the wrong benchmark for evaluating a scraping pipeline, because it measures general assistant reasoning with tool use, not extraction throughput.

A note on how easily these figures drift. The Hugging Face dataset viewer splits GAIA across an all-questions config and three per-level configs, so summing the rows on screen returns roughly double the real total. The paper says 466. Counting rows in a viewer is not reading a benchmark, and the same class of error is what puts inflated numbers into circulation.

WebVoyager pairs a multimodal agent with a Selenium-driven browser on real, live sites, with tasks drawn from 15 popular websites. Its most instructive design choice is the automated evaluator: scoring is done by a GPT-4V-based judge rather than a human, and the paper reports that this metric reaches 85.3% agreement with human judgment. That figure is the honest part. It also means roughly one in seven judgments diverges from a human's, so a reported 59.1% task success rate, which is what the abstract claims for WebVoyager itself, carries an error bar inherited from the grader. Model-graded evaluation scales, and it introduces a judge whose failure modes correlate with the system under test. WebVoyager is the wrong choice if you need scores you can defend to an auditor without also defending the judge.

Two more are infrastructure rather than benchmarks. BrowserGym, from ServiceNow Research, wraps MiniWoB++, WebArena, VisualWebArena, WorkArena, AssistantBench, WebLINX and others behind one Gymnasium-style Playwright API, so the same agent code runs against all of them. That is the practical answer to harness-dependent scoring: hold the harness fixed and vary the agent. Bananalyzer, from Reworkd, runs tasks against static MHTML snapshots of real sites through Playwright, with an AgentRunner interface you implement yourself. Static snapshots buy exact reproducibility at the price of ever testing a live defense. Both are wrong for teams who want a number today without writing code; they are right for teams who intend to keep measuring.

None of these will tell you which proxy network to buy. They are here because they demonstrate the properties a commercial benchmark usually lacks: a published task set, a scoring function you can read, a license, and a repository.

The checklist

Before a published benchmark changes your shortlist, get answers to these. If the report does not contain them, that absence is itself the finding.

1. What counted as success? One sentence, ideally with the check that produced it.
2. Which targets, and how many? Names, not categories. Per-target results, not just the aggregate.
3. When? A date range, not a publication year.
4. At what request rate and concurrency? And is there a second condition to compare against?
5. How many requests per target? Sample size determines whether a five-point gap between vendors means anything.
6. Retries: counted or excluded? This alone can move a success rate by double digits.
7. Who paid, and who participated? Named, on the page, not inferred from the logo grid.
8. Can it be rerun? A public harness turns an argument into an experiment.

Run the only benchmark that decides anything

No published benchmark answers your question, because your question is about your targets, your fields, your rate and your budget. Public reports narrow a shortlist. They do not replace a trial.

A workable in-house test is small. Take 200 to 500 URLs sampled from targets you actually scrape, including the ones you know are hard. Define success as a validated record, not a 200. Run the same set against each candidate on the same day, at the rate you plan to run in production, and record retries and cost per validated record alongside the success figure. Repeat the run a month later, because the second data point is where reliability starts to become measurable. If two vendors come out close, the multi-vendor failover route is usually more valuable than picking a winner.

Choosing which categories to trial in the first place is a separate exercise, covered in choosing a web scraping tool and, for the unblocking-specific decision, web unblocker vs scraping API. If your evaluation involves autonomous agents rather than fixed scrapers, the agentic automation category and the harnesses above are the better starting point.

Bottom line

Most published figures in this market are defensible inside the run that produced them and close to meaningless outside it. The numbers are usually fine; the argument wrapped around them is where the work is. Ask what success meant, which sites were hit, on what dates, at what rate, and who funded it. Publishers who answer inside the document rather than in a support email are uncommon enough that answering is itself a signal. Then run your own 300 URLs, because the only success rate that predicts your bill is the one measured on your targets.