The web's value exchange is collapsing. We're here to fix it.

AI crawlers scrape your content thousands of times for every referral they send back. Publishers are blocking AI bots in droves—75% block CCBot, 69% block ClaudeBot, 62% block GPTBot. But blocking isn't a business model.

Cloudflare's Pay-Per-Crawl launch proved what publishers actually want: compensation, not just control. The marketplace exists. What's missing is the playbook for pricing, negotiating, and maximizing revenue from AI content licensing.

AI Pay Per Crawl fills that gap. Not another blocking tool. The implementation guide for publishers transitioning from robots.txt blocking to monetization infrastructure—the Stripe Atlas for AI content licensing.

Cloudflare built the marketplace. We teach publishers how to use it profitably.

What we cover

→
Technical Implementation. RSL protocol, llms.txt specification, Cloudflare Pay-Per-Crawl configuration, crawler analytics setup.
→
Pricing Strategy. Per-crawl vs. per-inference vs. flat licensing models, industry benchmarks, calculating your content's training value.
→
Deal Negotiation. What Reddit, AP, and News Corp actually negotiated vs. what AI companies claim. Contract clause breakdowns.
→
AI Crawler Economics. Training data vs. retrieval data valuation, why Anthropic scraped 73,000x per referral vs. Google's 14x.
→
Enforcement. What happens when AI companies ignore payment standards, legal remedies, public pressure tactics.

AI Pay Per Crawl is a Scale With Search project.

Follow @b2bvic LinkedIn

Need this built for your site?

Victor Romo builds AI-ready search infrastructure for publishers and businesses.

See The Offer