robots.txt for AI Crawlers: Directive-by-Directive Configuration Guide

Your robots.txt is a note taped to the door asking visitors to behave. Well-run AI crawlers read the note and honor it. The badly behaved ones read it and keep walking, because nothing in the Robots Exclusion Protocol forces compliance. Treating that file like a lock is how sites end up surprised their content trained a model anyway.

Used correctly, though, it is the first and cheapest lever you have. This is the directive-by-directive guide: what the file can and can't do, the exact syntax that works, which tokens each AI operator answers to, and where you need a firewall to back the request with force.

• What Can robots.txt Control for AI Crawlers?
• How Does robots.txt Syntax Work for AI Bots?
• Which AI Crawler Tokens Need Separate Rules?
• Which AI Crawler Access Policy Should You Configure?
• How Do You Deploy and Enforce the Policy?
• How Do You Test and Maintain AI Crawler Rules?
• Key Recap
• FAQs

Quick Summary

• What this covers: configuring robots.txt for AI crawlers directive by directive, plus when to enforce with a firewall.
• Who it's for: publishers deciding which AI bots may read what, without breaking Google Search.
• Key takeaway: robots.txt is a voluntary request; pair it with Web Application Firewall rules when you need a real block.

What Can robots.txt Control for AI Crawlers?

The most expensive misconception in this whole topic is that robots.txt blocks anything. It does not. It expresses a preference that compliant crawlers choose to honor, and that distinction decides whether your content is actually protected or only politely labeled.

robots.txt controls crawler permissions by path: which user agents may request which parts of your site. The file must sit at the root of the domain, served at /robots.txt, or crawlers won't find it and will assume open access. It is a voluntary directive under the Robots Exclusion Protocol, not an enforced block. A well-behaved GPTBot obeys a Disallow. A scraper spoofing a browser ignores the file entirely, which is why enforcement matters further down.

Purpose is where AI crawlers diverge from the old model, and lumping them together costs you visibility or training exposure you didn't intend:

• Training crawlers gather content to build models. Blocking them keeps your text out of training data.
• Search-indexing bots feed answer engines. Blocking them can drop you out of AI search results and the referral traffic those results send.
• User-request fetchers act only when a person asks the assistant about your page. Blocking them stops those on-demand lookups.

Block the wrong purpose and you can vanish from AI answers while still feeding the training you meant to stop. The full operator inventory sits in the AI crawler directory, and the separate question of model-facing content guidance belongs to the llms.txt specification, not this file.

How Does robots.txt Syntax Work for AI Bots?

One stray character in this file can wall off your entire site from every crawler, including Google. The syntax is small, unforgiving, and worth getting exactly right.

A rule group starts with one or more User-agent lines naming the bots, followed by Allow and Disallow paths that apply to them. Disallow: /private/ blocks that path; Allow: /private/public-note.html carves an exception back out. Wildcards extend the reach: * matches any sequence of characters, and $ anchors the end of a URL, so Disallow: /*.pdf$ blocks every PDF. Precedence follows specificity: a bot obeys the most specific group naming it, and only falls back to the User-agent: * wildcard group when no named group matches it. A crawler with its own named group ignores the wildcard group completely, which trips up publishers who assume * rules stack on top of named ones.

The failure that blocks everything:

# BROKEN - a bare Disallow: / under the wildcard blocks all crawlers
User-agent: *
Disallow: /

That single slash tells every compliant crawler to stay off the entire site. Combine it with a forgotten Allow and you've deindexed yourself. A safe baseline instead:

# SAFE baseline - everyone allowed, PDFs and admin closed
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /*.pdf$

Sitemap: https://example.com/sitemap.xml

On the two extras: many AI crawlers honor Crawl-delay to space out requests, and Anthropic references crawl-delay support for ClaudeBot, though Google ignores it in favor of its own controls. The Sitemap directive is universally read and belongs at the file's end. Keep groups separated by blank lines; merging two User-agent blocks without a break is a common grouping error that silently misapplies rules.

Which AI Crawler Tokens Need Separate Rules?

Writing one rule for "AI bots" is impossible, because there is no single token for them. Each operator runs multiple named crawlers for different purposes, and you have to address them by exact string.

The token that saves publishers the most grief is Google-Extended. It governs Google's AI training and grounding and is completely separate from Googlebot, which crawls for Google Search. Disallow Google-Extended and your search ranking is untouched; disallow Googlebot and you've deindexed from Google entirely. Per Google's crawler documentation, these are independent tokens, and confusing them is the single most damaging mistake in an AI robots policy.

Verification still matters here, because a token in a header is only a claim. OpenAI publishes IP ranges for its crawlers, Anthropic references source-IP verification for ClaudeBot, and Google supports reverse DNS on Googlebot. A request wearing a name from an IP that doesn't back it is a spoof your robots file can't stop, which is where enforcement comes in. Keep this table current against the AI crawler directory, since operators add tokens regularly.

The Short Version: robots.txt expresses a preference compliant crawlers honor, so pair it with a Web Application Firewall when you need a block a scraper cannot ignore.

Which AI Crawler Access Policy Should You Configure?

Now the decisions. Most publishers want something more surgical than "all or nothing," and the protocol supports it if you write the groups carefully.

Allow AI search, block AI training. This keeps you visible in AI answers while keeping your text out of model training:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Google-Extended
Disallow: /

Block one crawler without touching Google Search. Note that Googlebot is never mentioned, so its access is untouched:

User-agent: PerplexityBot
Disallow: /

Path-level policy for public, private, and expensive content. Allow the free articles, close the paywalled and compute-heavy paths:

User-agent: GPTBot
Allow: /articles/
Disallow: /members/
Disallow: /search
Disallow: /api/

Give subdomains and staging their own robots.txt, because rules don't inherit across hosts; a staging.example.com with no file is wide open. Document every allow, block, and license decision by crawler purpose, so a policy review can reconstruct why GPTBot is blocked but OAI-SearchBot is allowed. That purpose-based matrix is also what feeds the AI content licensing models decision and, when you move to charging, the Cloudflare Pay Per Crawl setup.

How Do You Deploy and Enforce the Policy?

A perfect robots.txt that a scraper ignores protects nothing. Deployment gets the file live; enforcement is what gives it teeth against the crawlers that don't play along.

Deployment varies by stack:

• CMS: many platforms generate robots.txt from settings or a plugin; edit there, not by uploading a file the CMS overwrites.
• Static site: drop robots.txt in the public root so it builds to /robots.txt.
• CDN or reverse proxy: serve or rewrite the file at the edge, which lets you manage it centrally across origins.

Enforcement is a separate layer, because robots.txt is a request and some crawlers refuse it. A Web Application Firewall turns the preference into a block by matching user agent or verified-bot identity and rejecting the request outright. Cloudflare's own guidance draws exactly this line: robots.txt offers guidance, security rules enforce it. Cloudflare AI Crawl Control sits on top with per-crawler allow, block, and charge controls, robots.txt violation reporting, detection tiers, and WAF integration, so a crawler ignoring your file gets stopped at the edge instead of politely asked again.

For paid or protected content, rate limits and custom responses go further than a block. A 402 Payment Required response signals content is available for a fee rather than simply refused, and a 403 Forbidden hard-denies. Pairing a robots Disallow with a WAF rule that returns 402 Payment Required is the mechanical basis for charging crawlers rather than only blocking them. One caution on propagation: crawlers cache robots.txt, often for up to a day, so a change won't take effect instantly. Invalidate any CDN cache on the file after editing, and expect a lag before behavior shifts.

How Do You Test and Maintain AI Crawler Rules?

The costliest robots mistakes ship silently, because nothing errors, traffic quietly disappears. A test-and-maintain routine catches the damage before your rankings do.

Test before you deploy, by user-agent group:

1. Run each rule against the specific tokens it targets using a robots tester, checking that GPTBot is blocked where intended and Googlebot is still allowed everywhere it should be.
2. Confirm the file returns HTTP 200 at /robots.txt. A 404 means no rules apply; a 500 or a 401 can make the file ambiguous or unreadable, and some crawlers treat a persistent 5xx as "disallow all," which can accidentally deindex you.
3. Fetch the live file from a few locations to confirm the CDN is serving the version you edited, not a cached predecessor.

After deployment, monitor compliance from your logs. Server logs reveal whether a crawler is honoring the rules or hitting disallowed paths anyway, and repeated 200s on blocked paths from a verified crawler is a violation worth escalating to enforcement. This is exactly where log monitoring and robots policy meet, so a crawler that ignores the file shows up in the record you keep for licensing and incidents, alongside any RSL protocol implementation terms you publish.

Keep a rollback checklist so a bad edit doesn't linger:

• Save the previous robots.txt version before every change.
• After deploy, verify Googlebot and OAI-SearchBot still reach public content.
• If search access breaks, restore the prior file and purge the CDN cache immediately.
• Confirm recovery by re-fetching /robots.txt and re-running the tester.

Take Action: aipaypercrawl.com - Set your per-crawler policy, then point the reader to the pay-per-crawl setup guide.

Key Recap

• robots.txt is a voluntary request under the Robots Exclusion Protocol, not a security control.
• It must live at the domain root and applies by path and user agent.
• AI crawlers split into training, search, and user-request purposes; block the wrong one and you lose visibility or referrals.
• Address bots by exact token; Google-Extended is separate from Googlebot, and confusing them is catastrophic.
• Precedence favors the most specific named group; a bare Disallow: / under the wildcard blocks everyone.
• Deploy per stack, then enforce with a Web Application Firewall, Cloudflare AI Crawl Control, and 402 Payment Required or 403 Forbidden responses.
• Test by user-agent before deploy, watch logs for violations, and keep a rollback ready.

FAQs

Does robots.txt actually stop AI crawlers?

Only the compliant ones. robots.txt is a voluntary directive that well-behaved crawlers like GPTBot honor. Scrapers that ignore it need a Web Application Firewall or CDN rule to be actually blocked.

How do I block AI training but stay in AI search?

Disallow the training tokens like GPTBot and Google-Extended while allowing the search tokens like OAI-SearchBot. Purpose-specific rules keep your content out of training data without dropping you from AI answers.

Will blocking Google-Extended hurt my Google Search ranking?

No. Google-Extended governs AI training and grounding only. Googlebot handles Google Search and is a separate token. Disallowing Google-Extended leaves search ranking untouched.

Where does the robots.txt file have to go?

At the root of the domain, served at /robots.txt and returning HTTP 200. Subdomains and staging sites need their own file, because rules don't inherit across hosts.

How can I charge AI crawlers instead of only blocking them?

Pair a robots Disallow with a firewall rule returning 402 Payment Required, or use Cloudflare AI Crawl Control per-crawler charge controls. That signals paid access rather than a flat refusal.