robots.txt is quietly turning into a license. Your crawler still reads it as a yes/no

A License: line has started showing up in robots.txt files. Not Disallow, not Crawl-delay, but a directive that points at a document setting terms for what you may do with the pages underneath it. If your crawler uses a standard robots parser, it read that file, obeyed the Disallow rules, and skipped the license without noticing it was there.

That gap is what this post is about. For most of the web's history, collecting data meant clearing one barrier: the technical one. Blocks, fingerprinting, the proxies and unblockers you buy to get a clean response back. The RSL licensing standard adds a second barrier on a different axis, a licensing one, and the two no longer move together. A crawler can be fully robots-compliant and still walk straight past a license the publisher considers binding.

What the RSL licensing standard actually is

RSL stands for Really Simple Licensing. It is an open standard, published by the RSL Collective, that lets a publisher attach machine-readable licensing terms to content instead of a bare yes-or-no crawl rule. Where robots.txt says "you may fetch this," an RSL document says "you may fetch this, for these uses, under these terms, at this price."

The RSL Collective introduced the standard in September 2025 and declared RSL 1.0 an official industry standard on December 10, 2025. Its co-founders are Doug Leeds, co-CEO and former CEO of Ask.com, and Eckart Walther, a co-creator of RSS who chairs the technical steering committee. The parties around it are not fringe: Cloudflare, Akamai, Creative Commons, and IAB Tech Lab are involved on the standards side, and the Collective says more than 1,500 organizations have voiced support, among them the Associated Press, The Guardian, USA Today, Yahoo, Ziff Davis, O'Reilly, Stack Overflow, and Reddit.

What makes RSL relevant to a collection pipeline is where the license can live. The specification defines five channels for attaching one: a License: directive in robots.txt, an HTTP Link header with rel="license" and type="application/rsl+xml", an HTML <link rel="license"> element or an inline <script type="application/rsl+xml"> block, an <rsl:content> element inside an RSS feed, and embedded metadata in files such as EPUB, XMP, ID3, and PNG. The same terms can ride along on the request, the page, the feed, and the file.

It's a license, not a block

The reason this is more than a rebranding of robots.txt is that RSL separates rights by use, and it can attach a price.

RSL defines usage tokens that distinguish what a machine intends to do with the content. ai-train covers training or fine-tuning a model. ai-input covers feeding content into a model at run time, including retrieval-augmented generation and grounding. ai-index covers inclusion in an AI system's database. search covers building a conventional search index. ai-all and all are the broad catch-alls. A publisher can permit search while prohibiting ai-train, a distinction robots.txt has no vocabulary for.

Then there is payment. RSL supports a free license, attribution (visible credit and a working link), and several paid models: purchase for a one-time fee, subscription for recurring access, crawl for a per-access charge, training for a per-training-event fee, use for payment each time content contributes to an AI-generated output, and contribution for a good-faith monetary or in-kind payment. Amounts are expressed with an <amount currency> field, and the standard supports programmatic settlement through the x402 protocol, which revives HTTP's long-dormant 402 Payment Required status via an <accepts type="application/x402+json"> declaration. The signal is no longer allow or deny. It is a reservation of rights with a price attached.

Why a robots-compliant crawler still misses it

The trap sits in the parser. The robots.txt most crawlers read is governed by RFC 9309, the standardized Robots Exclusion Protocol. That spec defines allow, disallow, user-agent, and crawl-delay handling. It does not define a License: directive, and a conformant parser is under no obligation to do anything with a line it does not recognize. It reads the access rules, applies them, and moves on.

So a pipeline can pass every robots.txt check it makes and still ignore the RSL license sitting three lines below the Disallow block. The rel="license" header and the inline RSL script are equally invisible to a fetch layer that was only ever built to answer "am I allowed to request this URL." Nothing errors. Nothing logs. The crawl looks clean.

That is why the machine-readable part matters more than it first appears. A license buried in a website's terms-of-service PDF is something a publisher has to argue you should have found. A license expressed in a format with a published specification, attached to the file you fetched, is much harder to wave off as something you could not reasonably have seen. Machine-readable is the property that removes the "we didn't know" defense, whether or not your parser chose to read it.

Where this already has teeth: the EU

RSL is a format, not a law, and a format binds no one by itself. In the European Union, though, machine-readable reservation is already the switch that decides legality for a large class of collection.

Article 4 of the 2019 Copyright in the Digital Single Market Directive (2019/790) permits commercial text and data mining only where the rightsholder has not reserved those rights, and it specifies that for content made public online the reservation must be expressed in machine-readable form. A machine-readable "no" is what turns otherwise-lawful mining into infringement. The EU AI Act (Regulation 2024/1689) builds on that: Article 53(1)(c), in force since August 2, 2025, requires providers of general-purpose AI models to put in place a policy to identify and comply with those Article 4 reservations, including through state-of-the-art technologies. The accompanying GPAI Code of Practice, published in July 2025, asks signatories to run crawlers that read and respect robots.txt in line with RFC 9309.

RSL is one concrete way to express the reservation those rules already assume exists. This is an EU frame; in the United States the exposure runs through different theories, which earlier posts here have covered and this one will not restate. The point for a collection team is narrower: in at least one major market, ignoring a machine-readable reservation is not a matter of etiquette.

Does it bind a price monitor, or just AI trainers?

An honest reading matters here, because it is easy to over-claim. Most of RSL's design energy, and most of the legal pressure behind it, points at AI training and inference. The ai-train, ai-input, and ai-index tokens exist to govern exactly that, and a team running a nightly price scrape against a retailer is not training a foundation model. If a publisher's RSL document only reserves ai-train, it is not obviously speaking to your crawler at all.

The caveat is the all token, which is broad by design and does reach ordinary automated processing. Whether a publisher who asserts all can make it stick against a conventional scraper is an open question that will be answered target by target and, eventually, in court. This is a different axis from Cloudflare's pay-per-crawl approach, which meters declared AI crawlers at the network edge. RSL is publisher-authored rights expressed in the content, not access metering enforced by an intermediary. The two can apply to the same page and answer different questions.

What a collection team should do in 2026

None of this requires a panic, and none of it requires stopping. It requires reading a file you are already fetching more carefully.

• Read robots.txt past RFC 9309. At a minimum, detect and log the License: directive and any rel="license" header on your targets. You cannot make a decision about a signal you never captured. If you are unsure why a given site blocks or licenses the way it does, why scrapers get blocked is a useful backdrop.
• Classify your own use. Map what you actually do (search-style indexing, price monitoring, model training) onto RSL's tokens, so you can tell when a reservation is aimed at you and when it is not.
• Treat a pay-to-crawl offer as a buy-versus-fight call. For a high-value target that exposes an x402 price, paying the per-crawl fee may cost less than an escalating proxy and unblocker arms race against the same origin. For a low-value one, it may not be worth it. That is a procurement decision now, not just an engineering one.
• Fold RSL into diligence. When you evaluate a web data provider or a new target list, add "does this source publish a machine-readable license, and does it name our use" to the checklist.

The signal is changing character

For thirty years robots.txt was an etiquette file: advisory, unpriced, and easy to treat as optional. RSL does not repeal that file. It layers a licensing offer on top of it, in a format built to be read by the same crawler that already parses the access rules. The technical barrier and the licensing barrier have come apart, and clearing the first no longer says anything about the second. The teams that notice early will be the ones whose parser and whose legal posture both learned to read the new line before a publisher, or a regulator, made them.