Agents are winning the logged-in work, not the bulk crawl

OSWorld shipped a second version on 26 June 2026, announced on the v1 site. Reading the OSWorld 2.0 page on 27 July 2026, its own summary of the results says this: "Claude Opus 4.8 with maximum thinking and batched tool calls scores best but still completes only 20.6% of tasks at a 54.8% partial score; GPT-5.5 is far more token-efficient yet plateaus near 14%."

Twenty percent. On a benchmark built by academics from HKU, Salesforce Research, CMU and Waterloo, using execution-based checkers rather than a model grading itself. The original OSWorld paper put humans above 72% and the best model of that moment at 12.24%. Two years of model releases have roughly doubled the machine number and left the gap enormous.

Now read the other half of the market. Asteroid claims on its homepage, checked the same day, "37,000 records a month across 11 EHRs, zero interfaces built," and "16,000 a month across 80+ practices, 99.6% completion," and "Trusted by 100+ healthcare & operations teams." CloudCruise publishes a case study with a named customer, Trapeze, citing "5,000+ daily calls" and "100% workflow uptime over 3 months of continuous production." All of those are vendor self-reports on vendor pages, not audited figures, and should be read as marketing until someone else reproduces them. But the shape of the claim is consistent across the agentic automation category, and the companies making it are selling to buyers who can check.

So which is it. Are browser agents 20% reliable or 99% reliable?

Both, and the reason is not model quality.

The benchmarks measure the hard version

Halluminate's Web Bench is the cleanest illustration. Its repository, last updated 28 May 2025, describes 2,454 tasks across 452 live websites, with 1,580 of them (64.4%) classified as read tasks and the rest split across create, update, delete and file manipulation. Halluminate's write-up of those results reports most agents above 75% on the read half and the best agent, Skyvern 2.0, reaching 46.6% on the write half. That read-write gap does most of the explaining, and OSWorld 2.0 is measuring something harder still: full desktop and cross-application work, unconstrained, against checkers the agent cannot talk its way past.

An aside on a number that has been circulating. A best ClawBench model score of 33.3% appears in several secondary write-ups from mid-2026. ClawBench's own site did not render leaderboard entries when we fetched it on 27 July 2026, so we cannot confirm it and are not citing it as fact. We run no evaluations of our own, on this or anything else. The benchmarks category exists precisely so third-party numbers stay attributable to whoever produced them, and the guide on reading scraping benchmarks covers how much weight any of them deserve.

What benchmarks cannot capture is the deployment context. Every one of them measures an agent alone, on an unfamiliar site, with no retries, no human, no prior run to learn from, and no operator who narrowed the task first. That isolation is the right way to measure general capability and a poor way to predict production behavior, because nobody deploys under those conditions.

The workflows agents are actually winning have four properties

Look at what Asteroid and CloudCruise sell into. Eligibility verification against a payer portal. Referral intake. Credentialing checks across NPI and DEA registries. Pulling a patient record out of an EHR that has no export button. These share four properties, and each one moves the economics.

Volume per flow is low. A few thousand runs a month, spread across dozens of distinct portals. Script cost is paid per flow, agent cost is paid per run, so when the flow count is high and the run count per flow is low, the agent amortizes and the script does not.

Value per completion is high. An eligibility check that a person would spend eight minutes on has a floor price set by that person's hourly rate. A crawled product page is worth a fraction of a cent. The tolerable cost per attempt differs by three or four orders of magnitude.

A human was already in the loop. These processes had a person in them before the agent arrived, and Asteroid's approval checkpoint is that person, not a workaround bolted on to cover for the model. A one-in-three failure rate on a task where someone reviews the output is an efficiency change rather than a data quality incident.

And there is no API. The work was manual because the payer portal never shipped an interface, which means there is no cheaper, more reliable path sitting next to the agent.

Put those together and an agent that succeeds two times in three still beats the contractor it replaces, provided the failures are visible. That is the whole reconciliation. Benchmark and deployment agree about capability and disagree about what counts as good enough, and the deployment is right about its own economics.

Bulk collection is the opposite on every axis

Reverse all four properties and you get web-scale collection. One flow, millions of runs, value per record measured in fractions of a cent. There is no human in the loop by design, because a human in the loop is the thing you built the pipeline to remove, and there is usually a perfectly good deterministic path, because the page returns HTML.

Pointing an agent at that is buying a worse crawler at a higher price. The failure modes compound rather than cancel. Cost scales with steps and per-step token counts rise through a run, so a long traversal is superlinear. A 90% per-page success rate across a ten-page pagination chain is a 35% chance of a complete chain. And the characteristic agent failure is a confident wrong answer rather than a thrown exception, which a pipeline with no reviewer will happily persist.

The arithmetic is easy to anchor. Browserbase publishes its rates: on the Startup plan, listed at $99 per month when we checked on 27 July 2026, browser time runs "500 browser hours then $0.10/browser hr" and proxy bandwidth "5GB of proxies then $10/GB." That is before a single model token. Set that against a crawling framework on your own workers with a proxy contract underneath it, and the per-page comparison is not close at volume. Nothing about model progress changes it, because the gap is structural rather than a matter of accuracy.

There is also a diagnostic worth running before blaming the agent at all. Halluminate's stated reason for building a separate infrastructure benchmark was that browser infrastructure was observed to move agent accuracy by 25 to 50%. If runs are dying at the login page, that is an anti-bot system and a browser infrastructure problem. No model upgrade fixes it.

The decision rule

Four questions, answered before anything is procured.

Does the target expose an API, or return parseable HTML without a session? If yes, stop. Use the deterministic path. The 2026 stack post covers the layers.
How many runs per distinct flow? Thousands of runs on one flow is script work. Dozens of runs across dozens of flows is agent work.
What does one failed run cost? If a wrong record flows into billing, a regulatory filing or a customer-facing number without a reviewer, an agent's runtime-chosen path is a liability you will struggle to defend.
Does the task change state? Reads are the benchmarks' strong half. Writes are the weak half, and they are also the irreversible half. Writes need idempotency, a step budget and a human checkpoint, or they need to stay manual.

If the answers point at agents, the tooling splits by how much you want to own. Stagehand sits over Playwright and lets you replace only the steps that keep breaking. Browser Use hands the whole session to the model. Notte keeps credentials out of the model context in a secrets vault, which is the question your security review will actually ask. TinyFish bundles search, fetch, browser and agent behind one API. The guide to browser agents for hard targets goes through the cost formula and the hybrid pattern, which is the one most teams land on: let the agent discover the flow once, emit a script, run the script, keep the agent for repair.

What this means for a team collecting web data

Keep the crawl deterministic. Every page that returns HTML at volume stays on a framework and a proxy pool, and no agent goes near it.

Then audit the leftovers. Most teams have a list of targets they gave up on: a supplier portal behind a login, a licensing registry with no export, a partner dashboard someone copies into a spreadsheet every Monday. That list is the agent budget, and it is usually shorter and more valuable than expected.

For anything on it, write down the per-run cost ceiling and the observed success rate before committing. Cap the step count and cap the spend. Validate output against a schema so a null fails loudly instead of a plausible number passing quietly, and sample completed runs by hand to get a success rate you can put in the cost formula. The validating AI-extracted data guide covers the sampling side.

And treat any vendor number, including every figure quoted above, as a claim about that vendor's configuration until it survives your own targets. The benchmarks are measuring the general case. Your portal is not the general case, in either direction.