Public pages show content value.
We can see whether a site has research, reports, structured data, downloadable files, archives, or other resources that AI systems can use.
For publishers using Cloudflare
Turn AI website crawling into revenue.
Enter a domain. We find the pages AI crawlers may value. Connect Cloudflare later to see real crawler demand and set prices.
Example finding Research pages
- AI usefulness
- High
- Found pages
- 1,850
- Best next step
- Review
- Keep free?
- Probably not all
What this means This section may have AI value.
We would keep discovery pages free, then review deeper content for paid AI access.
The plain-English version
ClassifyToday separates content value from crawler demand.
Cloudflare shows demand.
After connection, we can compare those resource groups with actual AI crawler requests and explain the commercial opportunity with measured traffic.
Pricing rules turn insight into action.
Your team decides which groups stay free, which are charged, which are blocked, and which can move within approved pricing boundaries.
Cloudflare Pay Per Crawl, simply
Think of it as a checkout counter for AI crawlers.
Cloudflare is saying: if an AI crawler wants a page from your site, you can decide whether that crawler gets it for free, pays for it, or gets stopped.
An AI crawler asks for a page
A crawler visits your site the way a browser does, but it is collecting pages for an AI company or AI product.
Cloudflare checks the rule
Your site can say: allow this crawler, charge this crawler, or block this crawler.
If payment is needed, Cloudflare handles it
If the crawler is ready to pay, it gets the page. If not, Cloudflare can send back a payment-required response with the price.
You still need to choose smart prices
Cloudflare provides the gate and payment plumbing. ClassifyToday helps decide which parts of the site are worth charging for.
Based on Cloudflare's Pay Per Crawl beta documentation, which says the feature is currently in closed beta. Read Cloudflare's docs or Cloudflare's launch article.
How it works
A simple path from unknown AI usage to managed pricing.
Each stage answers a different executive question: what might be valuable, who is requesting it, and what should access cost?
Free public scan
We inspect public signals like sitemaps, robots.txt, structured data, page types, datasets, archives, update patterns, and information density.
You get a preliminary opportunity report that explains which content groups may be useful to AI systems.Cloudflare assessment
After you connect Cloudflare, we compare our content classification against actual AI crawler activity by URL pattern.
You see which valuable areas are already being requested, with scenario math based on observed demand.Managed pricing
ClassifyToday becomes the pricing layer for Cloudflare Pay Per Crawl. Your rules determine when content is free, charged, blocked, or dynamically priced.
Pricing changes happen in ClassifyToday, without treating Cloudflare as a manual spreadsheet of URL prices.What the first report says
Clear signals, careful claims.
Before Cloudflare is connected, ClassifyToday avoids revenue promises. It shows scenario economics so your team can understand the size of the opportunity without confusing examples for forecasts.
We can say
/research/* appears valuable. It has many resources, high information density, and machine-readable structure.
We do not say
You will make $4,300/month. Actual revenue depends on crawler demand, willingness to pay, and Cloudflare enforcement.
The next step
Connect Cloudflare. Then the assessment can compare resource groups with observed AI crawler requests.
Pricing control
Manage resource pricing in ClassifyToday, not one crawler setting at a time.
Cloudflare enforces Pay Per Crawl. ClassifyToday decides the price for each eligible request and keeps the history your team needs to understand what changed.
AI crawler Cloudflare ClassifyToday Worker Pricing decision Crawler-Price header
| Resource | Policy | Price | Reason |
|---|---|---|---|
| /research/* | Dynamic | $0.006 | High-utility analysis and recurring crawler demand |
| /data/* | Dynamic | $0.012 | Structured data with clear machine value |
| /products/* | Free | $0 | Discovery pages help crawlers reach paid resources |
| /archive/database/* | Fixed | $0.003 | Large reference collection with stable value |
Start with evidence
See which parts of your site might be worth pricing.
Questions executives ask
Enough detail to explain the model after one read.
The goal is not to make AI crawler pricing sound complex. The goal is to make the tradeoffs visible before a team changes access policy.
Can I charge AI crawlers to access my website?
Yes, if your site uses Cloudflare Pay Per Crawl and you have access to the feature. Cloudflare can require approved AI crawlers to pay for successful access to priced content.
What is Cloudflare Pay Per Crawl?
Cloudflare Pay Per Crawl is a Cloudflare AI Crawl Control feature that lets website owners charge AI crawlers for successful access to content.
How does Cloudflare Pay Per Crawl work?
An AI crawler asks for a page. Cloudflare checks the crawler and the site owner rule. If the crawler must pay, Cloudflare can return a payment-required response with the price or allow access when valid payment intent is provided.
What does HTTP 402 Payment Required mean for AI crawlers?
HTTP 402 means the AI crawler requested content that requires payment. Cloudflare can use that response to tell the crawler the content has a price.
What website content should I charge AI crawlers for?
Reports, datasets, research, archives, reference databases, documentation, and other information-dense resources are better candidates than homepages or navigation pages.
What website content should stay free for AI crawlers?
Discovery content should usually stay free: the homepage, sitemap, robots.txt, category pages, navigation pages, and pages that help crawlers find the deeper resources.
Is Cloudflare Pay Per Crawl a revenue forecast?
No. Pay Per Crawl creates a way to charge AI crawlers. Actual revenue depends on whether AI crawlers request your content, accept the price, and successfully retrieve the paid pages.
How do I know which pages AI crawlers might pay for?
Start with a public content scan to find high-value sections, then connect Cloudflare analytics to see which AI crawlers are actually requesting those sections.
Can Cloudflare charge different prices for different pages?
Cloudflare supports dynamic pricing through a response header from your origin or a Worker, so different content can use different prices instead of one zone-wide default.
Where does ClassifyToday fit with Cloudflare Pay Per Crawl?
Cloudflare handles crawler enforcement and payment infrastructure. ClassifyToday helps identify valuable content, recommend pricing policies, and manage pricing decisions by content group.
Is Cloudflare Pay Per Crawl available to everyone?
Cloudflare currently describes Pay Per Crawl as a beta feature, so availability may depend on Cloudflare account status, plan, and beta access.