Trusting AI-Extracted Data: Schema Design, Validation and Drift Detection

In 210 extraction attempts against deliberately wide schemas, the ExtractBench authors got valid JSON back about half the time and a fully correct record 4.6% of the time. Every vendor in the AI extraction category sells the step before that number: point a model at a page, describe the fields you want, get JSON back. None of them sell the step after it. Checking whether that JSON is right is left to you, and that is where data teams lose their time.

The reason this matters more with model-based extraction than with selectors is a difference in how the two fail. A CSS selector that stops matching returns nothing. You get a null, your pipeline sees a null, and someone gets paged. A model given the same broken page returns a value anyway, correctly typed and shaped exactly like the field you asked for. The pipeline is happy, the dashboard is wrong, and nobody notices until someone uses the number.

What follows is the layer between an extraction API and your warehouse, written for a team that already has a pipeline in production.

The failure that is specific to model extraction

Call it the plausible value. You ask for price from a product page. The page is a soft 404, or an anti-bot interstitial, or a redesign that moved the price into a lazy-loaded block the fetcher never saw. The model is holding a document with no price in it and a schema saying price: number. It returns 49.99.

Nothing downstream can tell that apart from a real 49.99. It passes type checking. It passes a range check. It is in the right ballpark because the model has seen a lot of product pages. The only thing that distinguishes it is that the string never appeared in the document, which is a fact your pipeline has thrown away by the time the row lands.

Published evaluations keep finding the same shape of problem. Cleanlab's structured-output benchmark, read on its blog on 27 July 2026, separates two numbers that vendors usually collapse into one. It defines field accuracy as the proportion of individual fields extracted correctly, and output accuracy as the proportion of samples for which every field is correct. On its PII extraction set the gap between them is enormous: field accuracy of 0.966 to 0.979, against output accuracy of 0.26 to 0.46. It runs a financial entities set alongside it where the same gap holds. Those are text documents rather than web pages, but the lesson transfers directly. A field-level accuracy number, which is what most vendors quote, tells you very little about how many of your rows are entirely correct.

The ExtractBench numbers at the top of this guide come from an arXiv preprint submitted on 12 February 2026 (Ferguson et al., arXiv:2602.12247, with co-authors including Contextual AI's Douwe Kiela), and they push the same point further on deliberately complex schemas. Across 210 extraction attempts it reports a 51% valid JSON rate and "an aggregate pass rate of 4.6%", while field-level accuracy among the extractions that produced valid JSON "ranges from 65% to 80%, with an aggregate of 72.9%". The best model in the set reached a 6.9% pass rate, and the paper reports 0% valid output from frontier models on a 369-field financial reporting schema. The authors are explicit that the valid-only figure is "computed on a biased sample" because the extractions that succeed tend to be the easier cases. Its corpus is 35 long PDFs rather than web pages, so treat the exact numbers as indicative. The direction is what matters: as schemas get wider and arrays get longer, whole-record correctness falls much faster than field correctness.

One more result worth knowing, because it changes something you control. NEXT-EVAL (arXiv:2505.17125, May 2025) benchmarked heuristic and LLM extraction of web data records across different input representations and found that flattened JSON representations of the DOM gave the best accuracy, an F1 of 0.9567, with less of what the authors call positional hallucination than slimmed HTML or hierarchical JSON. How you serialise the page before the model sees it is a lever, not a detail.

Schema design is the first validation layer

Most teams write the loosest schema that makes the extractor return something, then discover six months later that the schema was doing no work at all. Every field a string, every field optional, no enumerations. That schema cannot fail, which means it cannot catch anything.

Four decisions carry most of the value.

Type everything that has a type. A price is a number and a currency code, not the string "$49.99 USD". Parsing at the schema boundary means a garbled value fails there instead of failing in a warehouse query three weeks later. Tools that generate a schema for you, like Expand.ai, start you at a reasonable place, but a generated schema is a draft. It reflects one crawl of one page.

Enumerate anything with a fixed vocabulary. Availability, condition, currency, category, employment type. An enumeration is the cheapest hallucination detector in the stack, because a model that invents a value usually invents a plausible-sounding one outside your set. "Limited stock" arriving where you enumerated in_stock | out_of_stock | preorder is a signal you would never have got from a free-text field.

Decide required versus nullable per field, deliberately. This is the decision teams get wrong most often, and the consequences are not cosmetic. A required field turns a missing value into a loud failure at extraction time. A nullable field turns it into a null that flows to the warehouse and gets averaged over. Required is right for fields the page always has and the business depends on. Nullable is right for genuinely optional attributes, and every nullable field should have a monitored null rate, which is the subject of a later section. Making everything nullable is how a site redesign becomes invisible.

Version the schema and store the version on the row. Six months from now someone will ask whether a field changed because the site changed or because you changed the prompt. Without a schema version and a prompt hash on the row, that question has no answer.

The row-level checks

These run on every record, in the pipeline, before anything is written. They are cheap enough that there is no argument for sampling them.

Schema validation. Structural, and non-negotiable. Worth knowing: providers' native structured-output modes are not automatically the safer option. ExtractBench re-ran its full experiment through each provider's structured-output API and found overall validity dropped from 51% to 37%, with the best-model pass rate falling from 6.9% to 5.5%. Constrained decoding forces well-formed JSON, which is not the same as correct JSON, and on hard schemas it appears to cost accuracy. Validate the output yourself either way.

Grounding, or the substring check. For every extracted value that was supposed to be read rather than derived, confirm the string occurs in the document the model was given. Normalise first, since models reformat dates, strip currency symbols and collapse whitespace, so compare normalised forms and allow a fuzzy match on long free text. Keep a per-field grounding rate as a first-class metric. This is the check that has no analogue in selector-based scraping, and it is the one that catches the plausible value.

Derived fields need a different treatment. If your schema includes a total that the model computes, recompute it in code and compare. If it includes a normalised category the page does not literally contain, the check is set membership, not substring.

Cheap business rules. Range checks on numbers, a sanity check that a discount price is below a list price, a check that a date falls inside a plausible window, a check that a URL resolves to the domain you crawled. Each of these takes a line and each of them fires on real data more often than teams expect.

Fetch-quality checks that run before extraction. A surprising share of extraction errors are fetch errors wearing a costume. A 200 response containing a challenge page, a login wall or an empty shell is the input that produces the most confident wrong answers. Check document length, check for known interstitial markers, check that the final URL after redirects is still the URL you asked for. If your fetch layer is a scraping API or browser infrastructure provider, this is where its status codes and block signals earn their keep. The guide to why scrapers get blocked covers the failure modes that produce these documents in the first place.

Cross-source and cross-method agreement

Row-level checks catch values that were never on the page. They do not catch values that were on the page and were read from the wrong place, which is the other half of the problem. A price extractor reading a strikethrough list price instead of the sale price produces a grounded, typed, in-range, entirely wrong number.

Agreement checks are the practical answer, and you do not need to run them on everything. Pick the fields that carry money or drive decisions and check those.

Two extraction methods on the same document. Run a second extractor with a different architecture on a sample and compare. Diffbot is genuinely useful here because it is not an LLM pipeline. It applies computer vision and machine learning to the rendered page, an approach it has used since well before the current generation of models, so its errors are uncorrelated with a prompt-driven extractor's errors. Where Diffbot and a prompt-based tool like ScrapeGraphAI agree on a field, you have real evidence. Where they disagree, you have a review candidate. Diffbot is the wrong choice as your primary extractor if you need fields outside its supported page types, since its strength is article, product and discussion structures rather than arbitrary schemas, and it has no self-hosted option at all.

Two sources on the same entity. If the same product, company or listing appears on more than one site you crawl, disagreement between them is a signal about both. This is also where a commercial dataset from the data providers category is worth its price even if you do not use it as your primary feed: as a reference set for a few thousand entities, it calibrates everything else.

The same source over time. A field that changes on a page that did not change is an extraction error. Diffing your own previous run against the current one, gated on whether the page content hash changed, is the cheapest agreement check available and it requires no second vendor.

Keep a boring deterministic extractor in the stack for this purpose even if you never ship its output. A heuristic content extractor such as trafilatura costs almost nothing to run, has no per-call bill, and fails in the opposite direction to a model: it under-extracts rather than inventing. Running it alongside your primary extractor on a sample gives you a free second opinion whose errors are uncorrelated with the model's. Heuristics are not better than models here. They fail in a different direction, which is the whole reason the pair tells you anything.

Distribution monitoring, or catching the redesign

Most teams skip this, and it is usually the piece that pays for itself first. A site redesign does not throw an error. It produces a field that goes from 2% null to 30% null over a single run, or a price field whose median falls because the extractor moved to a different element, or a category field whose distribution collapses onto one value. Nothing in Scrapy or Crawlee will tell you, because at the HTTP level nothing went wrong.

What to record per source, per field, per run:

Two design rules make this work. Alert on change, not on absolutes, because every source has a different baseline and absolute thresholds either scream or stay silent. And scope alerts per source, because a 30% null rate on one domain averaged across two hundred domains is invisible.

Kadoa is the vendor that has built most explicitly around this idea. Its site, read on 27 July 2026, describes agents that "generate and maintain deterministic code and do not produce black-box LLM outputs", a self-healing layer where "when a workflow breaks, Kadoa detects it and fixes the code automatically" with "every fix logged so you can see what changed, when, and how it was resolved", grounding where "every value is source-grounded and audit-ready" and you can "trace any data point to see the exact page, paragraph, or cell it came from", and alerting "via Slack, email, or webhooks". That is the architecture this section argues for, sold as a product.

All of it is Kadoa's own description of Kadoa, and I have found no independent evaluation of the self-healing or grounding behaviour, so treat it as a design claim rather than a measured property. Kadoa is the wrong choice for a team that wants to start this afternoon: pricing is not published on the site, and the emphasised use cases are investment research, hedge funds, portfolio monitoring and quantitative finance rather than general-purpose web collection. If your targets are e-commerce listings at high volume, you are not the buyer it is built for.

If you are not buying that, the same monitoring is a few hundred lines over your run metadata plus whatever your team already uses for metric alerting. The build-versus-buy line is roughly at the number of distinct sources you monitor. Below twenty, build it. Above two hundred, the maintenance argument for buying gets strong.

Provenance

Every row should be traceable to the bytes it came from. Concretely, store alongside each record:

• The fetch timestamp and the final URL after redirects, not the requested URL.
• A content hash of the document the extractor received.
• A pointer to the stored document itself, compressed, in object storage.
• The extractor identifier and version, and the prompt or schema version.
• The model identifier where the extractor exposes one.
• Any vendor-supplied confidence or grounding output.

The cost objection is real at high volume and the standard compromise is a retention window: keep full documents for thirty to ninety days, keep hashes and metadata forever. That is enough to reproduce a recent dispute and enough to prove a document changed even after you have deleted it.

Provenance is what makes every other check in this guide actionable. Without it, a grounding failure tells you something is wrong but not what. With it, you pull the exact document, look at it, and know within a minute whether the model erred or the site moved.

Vendors vary a lot here, and the axis that matters is whether you control the fetch layer. FetchFox is open source and self-hostable, so you can persist exactly what you want and for as long as you want; the tradeoff is that it is a small project with a modest following, which makes it the wrong choice where procurement wants a vendor with a track record. Parsera has the same self-hosting property through an open-source Python core built on Playwright, and the same caveat: there is little independent coverage of the company, so longevity is harder to judge than for better-documented competitors. Fully hosted services give you less. JigsawStack routes scraping, search and OCR through one API built on small purpose-built models, which is convenient if you need all three, but web scraping is one endpoint in a broader suite rather than the focus, so deep multi-page crawling and heavy anti-bot work are not what it is for.

Expand.ai is worth reading closely as an example of how this gets marketed. Its site, read on 27 July 2026, promises "high quality with back checking" and states that "all data is checked and traced back to the source, making hallucination impossible". Treat that last clause as marketing rather than an engineering guarantee. Source-tracing meaningfully reduces invented values, which is exactly the grounding check this guide recommends, but it does not make hallucination impossible, and no vendor can make that true of a model pipeline. Y Combinator's directory lists Expand.ai as an active Summer 2024 company in San Francisco, founded by Tim Suchanek, with a team size of 2. I found no independent evaluation of the back-checking claim. The team size is the relevant risk for anyone putting a production pipeline on it.

Sampling for human review, stratified

Human review is the only check that catches errors your rules did not anticipate, and it is the most expensive thing in the layer. Spending it well is mostly a sampling problem.

Random sampling is the wrong default. If 95% of your corpus is one page template that extracts cleanly, random sampling spends 95% of your reviewer's attention confirming that. Stratify instead, and allocate a fixed weekly budget across strata:

That last row is the one teams cut and should not. Without a random control slice you learn about the errors your checks already flag and nothing about the errors they miss, and the second number is the one you actually want. It is also what turns review into a measurement: the error rate in the random stratum, tracked over time, is your honest quality figure. Everything else is triage.

Review at the field level, not the row level. "This row is wrong" is not an actionable label. "The price field on this template reads the list price" is.

Where the tools fit, and where they leave you

A note on the last two rows, since they are the ones people underestimate. When an agent navigates to reach the data, the failure mode expands: the agent can land on a different page than intended and extract that page perfectly. Grounding will pass, because the values really were in the document. What catches it is checking the final URL and page identity against what you asked for, which means your provenance record has to include the URL the extractor actually landed on. AgentQL sits in this category. It comes from TinyFish, and its repository, checked on 27 July 2026, is MIT-licensed and describes "a query language and Playwright integrations for interacting with elements and extracting data". It replaces CSS and XPath selectors with a query language, which is the point of it, and at 1,429 stars it is a much smaller community than the prompt-driven libraries below. It is the wrong pick if you want managed scraping infrastructure, because it gives you browser automation and expects you to bring proxy rotation and anti-bot handling yourself. The multi-vendor failover guide covers that side.

ScrapeGraphAI is the most accessible starting point of the prompt-driven group. Its Python library is MIT-licensed with 28,664 GitHub stars as of 27 July 2026, and there is a hosted API alongside it. Its pricing page, read the same day, lists one-time credit packs from $5.00 per 1,000 credits, falling to $4.00 and $3.00 per 1,000 at larger sizes, with monthly plans from $20 for 10,000 credits, and states that "all requests are SOC 2 Type II compliant". That last point is worth confirming directly if it matters to your procurement, since our directory entry records only a Type I attestation and prices and attestations both move. The structural caveat is inherent to the approach: extraction runs through a model rather than fixed field mappings, so output can vary run to run in ways a selector-based scraper does not. That is exactly why the grounding and distribution checks above are not optional if you build on it.

Vendor-run evaluations should be read with the obvious discount, and several extraction vendors publish one. A vendor measuring itself on a corpus it selected, with prompts it wrote, is not worthless, but it is not a number to plan capacity or accept risk against either. Ask which documents were in the set, who annotated the ground truth, and whether the reported figure is per field or per whole record, since the ExtractBench and Cleanlab results above show how far apart those two can sit. Third-party work belongs in the benchmarks category for a reason.

Build it in this order

If you are starting from a pipeline with no validation layer, the order that gets the most value soonest:

1. Provenance first. Store the document, the hash, the final URL and the extractor version. Nothing else is diagnosable without it, and it is the one thing you cannot add retroactively.
2. Schema tightening. Types, enumerations, and one deliberate pass over required versus nullable. A day of work, and it converts a class of silent failures into loud ones.
3. The grounding check. A substring test with normalisation, plus a per-field grounding rate. This is the highest-value single check and it is maybe fifty lines.
4. Distribution monitoring. Null rate and value distribution per source per field, alerting on change against a trailing window.
5. Stratified review. Start with a hundred rows a week across the strata above, including the random control slice.
6. Cross-method checks last. They cost the most, in a second vendor and a second bill, so add them for the fields that justify it once you know from steps 3 to 5 which fields those are.

None of this is exotic. It is the same discipline any data team applies to an internal ETL job, applied to a source that lies more confidently than a database does. The reason it keeps getting skipped is that model extraction demos so well: the JSON comes back clean on the first page you try, and clean JSON on one page is indistinguishable from clean JSON on a hundred thousand.

If you are still choosing the extraction layer itself rather than the checking layer, the guide to choosing a web scraping tool covers the category-level decision, and the 2026 stack overview covers how these pieces fit together. If you are evaluating hosted no-code options specifically, the deployment models guide is the relevant one. Whichever you land on, budget for the layer after it. The extraction is the part someone will sell you.