Data Parsing

Data parsing is the step that turns a fetched response into typed records. The fetch stage gives you bytes; parsing decides that this string is a price of 24.99 in EUR, that one is a publication date, and this block is navigation chrome to be thrown away. Four approaches dominate, in rough order of brittleness.

Regular expressions over raw text are the fastest to write and the first to break, because they encode assumptions about whitespace and attribute order that no HTML author promised to keep. Tree-based selection with CSS or XPath, the model behind Beautiful Soup and Nokogiri, is the default for good reason: it survives reformatting and reads like the page structure. Embedded structured data is better still when it exists. Schema.org JSON-LD blocks, Open Graph tags and the JSON payload that a client-rendered page uses to build itself are all already typed, already the publisher's own field names, and far more stable than the markup wrapped around them. Model-based extraction against a schema, offered by Diffbot and by the LLM extraction modes in newer scraping APIs, tolerates redesigns that would break every selector, at the price of per-page inference cost and non-deterministic output that needs validation.

Selector breakage on site redesigns is the dominant ongoing cost of any scraping operation, and it is the reason vendors sell parsed output at a premium over raw HTML. A crawl of 40 targets does not break all at once; it breaks one target at a time, quietly, and the field that vanished still returns a valid-looking empty string. Parsing therefore needs schema validation and per-field null-rate alerting more than the fetch stage needs uptime monitoring. Services like Scrapfly and Diffbot absorb that maintenance into a subscription. Parser leniency compounds it: real-world HTML is malformed and different parsers recover from it differently, producing different trees from the same bytes. The Beautiful Soup documentation gives the canonical example, an empty anchor followed by a stray closing paragraph tag. lxml discards the dangling tag and wraps the anchor in html and body elements, html5lib invents a matching opening paragraph and nests it inside the anchor, and Python's built-in html.parser adds no structure at all. Since the document is invalid, the docs note that none of these is the correct answer, and they recommend naming your parser explicitly so results do not shift between machines.

The decision is arithmetic. Estimate redesign frequency per target against record volume per month. High volume on stable targets favors selectors you maintain. A long tail of targets that each redesign once or twice a year favors buying pre-parsed output, because the cost of a selector engineer scales with target count while the vendor's does not.