Cloudflare Pay Per Crawl, Explained Simply

Cloudflare Pay Per Crawl is designed for charging approved AI crawlers for successful access to website content. Whether a specific crawler can be charged depends on Cloudflare support, crawler identification, your Cloudflare configuration, and whether that crawler participates in the paid access flow.

You use Cloudflare AI Crawl Control and Pay Per Crawl. At a high level, you enable Pay Per Crawl, set a price, choose which AI crawlers are allowed, charged, or blocked, and then monitor crawler activity.

It is a payment gate for AI crawlers. Human visitors still browse normally, but AI crawlers that want priced content may need to show they are willing to pay before Cloudflare lets them access it.

Cloudflare checks whether the crawler is allowed, charged, or blocked. If the crawler must pay and has not provided valid payment intent, Cloudflare can return HTTP 402 Payment Required with the price.

It means the AI crawler requested content that requires payment. The crawler did not receive the paid page yet; it received a response telling it payment is required.

The crawler-price header tells an AI crawler how much the content costs. For example, it can tell the crawler that access costs a specific amount in USD.

crawler-exact-price is a request header an AI crawler can send when it agrees to pay the exact price requested for the content.

crawler-max-price is a request header an AI crawler can send to say the highest price it is willing to pay for a piece of content.

crawler-charged is a response header that tells a paying crawler how much was charged for a successful paid content request.

Cloudflare documentation says error responses are not billed. Charging is tied to successful content access, not failed requests.

Cloudflare documentation says AI crawler operators should expect each successful paid access to incur the site owner price, including re-crawls.

Yes, with dynamic pricing. A site can use an origin response or Cloudflare Worker to return a crawler-price header for specific content.

When dynamic pricing is enabled, Cloudflare can use a crawler-price header from the origin or a Worker instead of the zone default price for that request.

Cloudflare can send cf-pay-per-crawl to the origin so your server or Worker knows whether the request is in Pay Per Crawl pricing mode.

Discovery pages should usually stay free: the homepage, sitemap, robots.txt, navigation pages, category pages, and other pages that help crawlers find paid content.

Research, reports, datasets, archives, reference databases, documentation, downloadable files, and structured pages are stronger candidates for paid AI crawler access.

robots.txt is an instruction file. It is not the same as enforcement. Cloudflare AI Crawl Control and Pay Per Crawl provide control and monetization tools beyond robots.txt.

Block crawlers when you do not want access at all. Charge crawlers when access could be acceptable if there is a value exchange. Some discovery pages should remain free.

Not automatically. Pay Per Crawl creates a payment mechanism, but revenue depends on crawler demand, crawler willingness to pay, your prices, and which content you charge for.

There is no universal price. Cloudflare documents a minimum price, but the right test range depends on content value, uniqueness, structure, and actual crawler demand.

Cloudflare AI Crawl Control analytics can show which AI services access your content, request patterns, paths, and crawler activity. A public scan alone cannot measure actual crawler demand.

Use a content value scan to identify high-value resources, then compare those groups with Cloudflare AI crawler analytics to see whether crawlers are actually requesting them.

For Cloudflare Pay Per Crawl, yes. The feature is part of Cloudflare AI Crawl Control and relies on Cloudflare for crawler enforcement and payment infrastructure.

Cloudflare currently describes Pay Per Crawl as a beta feature, so access may depend on beta availability, plan, and account status.

Yes. Cloudflare documentation says Workers can be used to include crawler-price headers in responses, which allows pricing decisions outside the origin application.

ClassifyToday scans your site, identifies content groups that may be valuable to AI crawlers, recommends what should be free or paid, and helps manage pricing logic on top of Cloudflare enforcement.