What is Nokogiri?
Nokogiri is an open-source, MIT-licensed Ruby library for parsing, searching, and manipulating XML and HTML documents. It ships with DOM, SAX, and Push parsers for XML, HTML4, and HTML5, supports document querying through XPath 1.0 and CSS3 selectors, and handles XSD schema validation, XSLT transformation, and a builder DSL for generating markup. It requires Ruby 3.2+ or JRuby 10.0+, and since it only parses static markup, JavaScript-rendered pages need a separate HTTP client or headless browser in front of it.
Our verdict
Nokogiri is the standard choice for Ruby developers who need to parse and query HTML or XML. It has 6.3K GitHub stars, 8,000+ commits, and corporate sponsors like Sentry and Gusto backing ongoing maintenance, though it does no fetching or JS rendering itself. Pair it with an HTTP client such as Mechanize (built on Nokogiri) or a headless browser for dynamic pages.
Categories:
Crawling frameworks give engineering teams full control over their pipeline: where it runs, how it scales, and what it collects. The trade is that anti-bot work and infrastructure become your problem rather than a vendor's, which is why they pair well with a proxy network.
