Cloudflare BotBase Makes Bot Identity Operational

Cloudflare added an operator side to BotBase, its directory of known bots and agents. The useful change is not just another dashboard tab. It gives bot operators a place to submit a bot, track review status, edit submissions, and declare how the bot behaves and uses content.

For teams building agents, crawlers, retrieval tools, or AI assistants that fetch public web pages, this is a clear signal: bot identity is becoming an operations surface, not a footer in a user-agent string.

What Changed

Cloudflare says BotBase for Operators now lives in the Cloudflare dashboard under Protect & Connect -> Application Security -> BotBase. It includes a bots directory, a submission form, and submission history.

The submission history is the most practical piece. Operators can see whether a submission is waiting for review, accepted, or rejected. If a submission is rejected, Cloudflare says the dashboard shows the reason and the steps needed to fix it.

Operators can also edit a submission after it has been created. That matters because bot identity is not static. A crawler may move its IP list, change verification methods, update its user-agent pattern, or shift from basic crawling to agentic page access.

Cloudflare also says the review path now includes automated checks for duplicates, user-agent specificity, and verification methods such as IP lists, reverse DNS, and Web Bot Auth signatures.

Why We’re Paying Attention

Most small teams do not think of themselves as bot operators. They think they are building a research assistant, a support copilot, a RAG ingestion job, a price monitor, or a workflow agent.

From the receiving website’s point of view, that traffic is still automated traffic.

That means the team has to answer a few questions before the first production crawl:

Who operates this bot?
What domains, IP ranges, or infrastructure does it use?
What user-agent identifies it clearly?
Does it index, summarize, train, transact, monitor, or act on behalf of a user?
Does it respect robots.txt and newer content-use signals?
Can a site owner verify that a request really came from the claimed operator?
Who responds when a site blocks, rate-limits, or questions the traffic?

Cloudflare’s update does not solve those questions for every website. It does make the direction obvious. Automated traffic is moving toward declared identity, declared behavior, and cryptographic or infrastructure-backed verification.

How It Works

BotBase already gives Cloudflare customers a way to browse known bots and agents, see behavior-based classifications, and target specific bots in security rules. Cloudflare Radar also exposes a public bots and agents directory for tracked bots.

The operator launch adds the missing feedback loop.

Instead of submitting a bot into a black box, an operator can now see the submission record. Instead of resubmitting from scratch when details change, the operator can update the existing submission. Instead of forcing one broad label, the form asks for a more practical description of what the bot does, how it uses content, and who is actually operating it.

The content-use part is especially important for AI systems. Cloudflare points to the Content Signals model, where a site can express preferences such as allowing search indexing while disallowing AI training. A bot that only fetches a page to answer a user’s live question is different from a bot that stores pages for model training. Treating both as the same kind of crawler is too blunt for where the web is going.

For verification, Cloudflare’s Web Bot Auth documentation describes an Ed25519-based HTTP message signature approach. In that model, a bot publishes a key directory at /.well-known/http-message-signatures-directory, registers the bot and key directory, and signs requests after verification. Cloudflare can then check that a request came from the registered bot instead of only trusting an easily copied user-agent.

A Small Useful Test

This post does not need a sample repo. The right takeaway is an operating checklist, not a toy crawler.

Before shipping an agent or crawler that touches third-party sites, write down this record:

Bot readiness record

Name:
Operator:
Purpose:
User-agent:
Contact URL or email:
Production domains:
Outbound IP ranges or provider:
Verification method:
Content access pattern:
Content retention:
Training use:
Robots.txt behavior:
Rate limit policy:
Block/abuse response owner:
Last reviewed:

Then test the public parts from outside your own network:

curl -I https://example.com/.well-known/http-message-signatures-directory
curl -A "YourBot/1.0 (+https://example.com/bot)" https://target-site.example/

The first command only applies if you use a key directory. The second command is a simple sanity check: your user-agent should be specific, stable, and tied to a page where site owners can understand what the bot does.

If the bot is important enough to run every day, it is important enough to have a clear identity page and an owner.

Cost And Operational Notes

BotBase itself is part of Cloudflare’s bot management ecosystem. Cloudflare’s docs describe BotBase availability for Enterprise Bot Management customers. The Radar directory is public, but operating inside the Cloudflare dashboard is not the same as a free standalone crawler registry.

That cost detail should shape expectations. A small team should not build a dependency on one vendor’s dashboard being present everywhere. The durable work is vendor-neutral:

Keep the bot’s public identity page current.
Use a specific user-agent that does not collide with other traffic.
Publish verification material when you can support it.
Separate search, reference, training, monitoring, and user-directed actions in your own logs.
Store enough crawl telemetry to answer site-owner questions later.
Treat blocks and rate limits as feedback, not just errors to route around.

The operational risk is reputation. If an agent fetches too aggressively, hides behind generic infrastructure, or copies another user-agent, it can get the whole product treated as hostile traffic. Identity work is cheaper before that happens.

What We’d Watch Next

The next useful step would be wider interoperability. Cloudflare is one large network, but bot operators need a repeatable identity pattern that works across CDNs, publishers, application firewalls, and independent sites.

Web Bot Auth is worth watching because it moves bot identity away from claims that can be copied and toward signatures that can be verified. Content Signals is worth watching because it gives operators a more precise vocabulary than “allowed” or “blocked.”

For now, the practical move is simple: if your product sends automated traffic to the web, treat bot identity as part of release readiness. The teams that can explain and prove what their agents are doing will have an easier time staying allowed.