Cloudflare Bot Preference Sync Keeps AI Crawler Policy Honest
Cloudflare's Bot Preference Sync turns AI bot policy into a generated robots.txt block, reducing drift between what a site says to crawlers and what it enforces at the edge.
Cloudflare announced Bot Preference Sync, a feature that keeps a site’s robots.txt aligned with the AI bot settings already configured in Cloudflare.
That sounds administrative, but it fixes a real operations problem. A team may block training crawlers at the edge while an old robots.txt still says something different, or publish a crawler preference that the enforcement layer does not match. Bot Preference Sync makes the public preference and the edge policy come from the same decision.
What Changed
Cloudflare now lets site owners manage AI bot behavior by category: Search, Agent, and Training. Bot Preference Sync reflects those choices into robots.txt.
If the zone already has a robots.txt, Cloudflare says the generated block is prepended to the existing file, so existing directives remain in place. The generated section is bounded by comments, and the bot list is updated from Cloudflare’s tracked bot data.
Cloudflare’s example policy is simple: allow Search, allow Agents, and disallow Training. With sync enabled, the site publishes a block shaped like this:
# BEGIN Cloudflare Bot Preference Sync
User-agent: TrainingBot1
User-agent: TrainingBot2
User-agent: TrainingBot3
User-agent: MixedUseBot-Extended
Disallow: /
# END Cloudflare Bot Preference Sync
The important part is not the exact bot names in the example. It is the source of truth. The team sets the policy once, and Cloudflare makes the public crawler preference match.
Why We’re Paying Attention
AI crawler policy has become more nuanced than “block all bots” or “allow all bots.” A software company may want documentation indexed by search-style AI systems. A publisher may want search visibility but not model training. An internal app may want agents blocked everywhere because the pages were never meant for automated browsing.
Those are different policies, and they are easy to misconfigure when robots.txt, CDN rules, bot management settings, and custom firewall rules drift apart.
Bot Preference Sync is useful because it treats crawler preference as part of production configuration. For small teams, that means fewer hidden policy files and fewer “we thought we blocked that” moments.
How It Works
Cloudflare classifies AI bot behavior into three use cases:
- Search: crawlers that collect or index content so an answer system can retrieve it later.
- Agent: automated activity acting on a person’s behalf in real time, such as chat fetch bots and browser-use agents.
- Training: crawlers that use content to train or fine-tune a model, including mixed-purpose crawlers used for both training and search.
For Search and Agent traffic, Cloudflare’s options remain Allow, Block on pages that serve ads, or Block everywhere. For Training, Cloudflare is introducing a Disallow preference that writes a no-training signal into robots.txt for cooperating crawlers while still allowing transparent mixed-use crawlers to keep search access.
That last detail matters. robots.txt is a preference protocol, not an authorization layer. RFC 9309 describes it as a way for service owners to control how crawlers may access resources, and it explicitly says those rules are not access authorization. Enforcement still has to happen somewhere else when the risk demands it.
A Small Useful Test
This post does not need a sample repo. The practical test is to compare your declared crawler policy with your actual enforcement path.
Run this before enabling any automated sync:
AI crawler policy check
1. Fetch the current production robots.txt.
2. List every crawler or category you intend to allow, block, or disallow.
3. Compare that list with CDN, WAF, bot management, and application rules.
4. Decide whether Search, Agent, and Training should have different treatment.
5. Check whether ad-supported pages need stricter defaults than the rest of the site.
6. Note any crawler-specific business exceptions before turning on category-wide sync.
7. After sync, fetch robots.txt again and confirm the generated block matches the intended policy.
For a Cloudflare-hosted site, the smoke test can be as plain as:
curl -fsSL https://example.com/robots.txt | sed -n '1,80p'
You are looking for policy agreement, not a clever script. The public file, dashboard setting, and enforcement layer should tell the same story.
Cost And Operational Notes
Cloudflare says Bot Preference Sync will be available to all customers, from Free to Enterprise, and can be turned on or off. New customers will have sync on by default, but non-publisher zones will not have blocks or disallows added automatically at onboarding. Publisher-style sites that say they monetize pages with ads can get Training set to Disallow by default.
There are a few operational edges to keep in mind:
- Category-wide policy is not custom logic. Cloudflare says Bot Preference Sync does not read individual custom rules with more complex conditions. If you have crawler-specific contracts or exceptions, review those before enabling sync.
- Prepending changes precedence conversations. Existing directives remain, but the generated Cloudflare block appears first. Test the final file as crawlers will read it, not as separate fragments.
- Disallow is not the same as block. Cooperating crawlers may honor
robots.txt; hostile or careless traffic still needs edge enforcement. - Bot identity changes over time. The value of a managed list is that it can update. The risk is that policy changes become less visible unless someone reviews the generated file periodically.
For most teams, the right rollout is narrow: document the intended policy, enable sync on one zone, fetch the resulting robots.txt, check logs for blocked and allowed crawler traffic, then expand.
What We’d Watch Next
The next useful step is better auditability. Site owners need to know not just which bot was blocked, but which policy decision caused it: Search allowed, Agent blocked on ad pages, Training disallowed, or custom rule override.
This is where AI bot management becomes more like access control. The setting is only half the work. The operator still needs a readable trail from business intent to public preference to edge enforcement to traffic evidence.
Bot Preference Sync is a practical move because it removes one common source of drift. It does not settle the whole AI crawler debate, but it gives teams a cleaner place to start.