A potential customer asks an AI assistant to find a replacement part your company sells. You want your product page to appear. You may even want the assistant to check whether the item is in stock.
Then the assistant tries to change the delivery address on an existing order.
Those are three different jobs. Your website should not treat them as one kind of “AI traffic.”
Search engine optimisation taught companies to make pages discoverable. The next job is more operational: decide what machines may discover, what they may retrieve on demand, and what they may change.
Cloudflare’s current bot controls separate AI traffic into three behaviours: Search, Agent and Training. Search collects or indexes content to answer questions later. Agent traffic acts in real time for a person, including chat fetchers and browser-use agents. Training traffic collects content to train or fine-tune a model.[1]
That distinction is useful beyond any one security product. It gives an SME a practical starting point for an agent-access policy.
Blanket rules destroy value in both directions
“Block all AI bots” sounds safe. It can also make your business harder to discover through AI-assisted search and buying journeys.
“Allow AI bots” sounds open. It can expose routes and forms to automated use that they were never designed to handle.
A product description and an account-change form are both web pages, but they carry different consequences. One helps a buyer understand an offer. The other can change a customer record, trigger fulfilment or create financial exposure.
The policy should follow the consequence of the action, not the label attached to the visitor.
Cloudflare’s documentation makes this difference visible at the traffic layer, but classification is not the same as authorisation. A bot label may help decide how to handle a request. It does not prove who the user is, whether the user owns the account, or whether the requested action is allowed.[1]
That requires application controls.
Zone 1: Discover
The discover zone contains public information that you want people and machines to find.
Typical examples include:
- product and service pages;
- opening hours and locations;
- public course information;
- FAQs and support documentation;
- public pricing or pricing principles;
- articles and case studies.
The business objective is reach. The main questions are whether the information is current, whether machines can parse it correctly, and whether the resulting answer leads people back to a useful next step.
Access can be broad because the content is already public. That does not mean “unlimited.” Rate limits, bot rules and abuse monitoring still matter. But the default posture can be discoverable unless there is a clear reason to restrict it.
Assign an owner for each important content family. If an agent surfaces last year’s price or an expired course date, the problem may not be the agent. It may be that nobody owns the source page.
Zone 2: Retrieve
The retrieve zone lets an agent fetch current information for a user without changing business state.
Examples include:
- checking appointment availability;
- retrieving a quotation status;
- reading an order status;
- checking stock by location;
- pulling a customer’s service entitlement;
- retrieving an invoice copy.
This zone is more useful than public discovery and more sensitive. Some requests can remain anonymous, such as checking general appointment slots. Others require identity because the answer belongs to a specific customer.
Design retrieval as a controlled interface, not as permission to browse everything.
Return only the fields required for the task. Separate public availability from customer-specific records. Apply rate limits. Log which identity requested which data. Give the user and support team a way to revoke access.
For many SMEs, retrieval is the best first agent integration. It removes repetitive checking without giving an external agent permission to alter orders, accounts or money.
Zone 3: Transact
The transact zone changes business state.
Examples include:
- submitting or accepting a quotation;
- booking or cancelling an appointment;
- changing an order or delivery address;
- updating an account record;
- requesting a refund;
- creating a purchase or payment instruction.
This zone needs enforceable identity, permissions and approval rules. A robots.txt entry is advisory. A bot category is classification. Neither replaces authentication, authorisation or validation inside the application.
A transactional agent should receive no more authority than the user it represents. High-consequence actions should add friction deliberately: confirmation screens, one-time approvals, value thresholds, dual approval or a human exception queue.
The rule is simple: if the action would concern you when performed by a rushed employee, it should concern you when performed at machine speed.
A practical decision matrix
Use this table for every machine-facing route:
| Decision | Discover | Retrieve | Transact |
|---|---|---|---|
| Business value | Reach and understanding | Faster answers and service | Completed work |
| Data sensitivity | Public | Public or customer-specific | Customer, operational or financial |
| Identity | Usually optional | Required for private data | Required |
| Allowed action | Read public content | Read approved fields | Change approved state |
| Rate limit | Abuse-based | User, agent and route based | Strict, risk-based |
| Human approval | Rare | Exceptions | Threshold or consequence based |
| Logging | Traffic and content freshness | Identity, fields and result | Full action, actor, before/after state |
| Revocation | Bot or route rule | Token, user or integration | Immediate access and transaction stop |
The table is not a technical specification. It is the business decision that should come before one.
The owner of a quotation workflow may accept automated status checks but require a salesperson to approve discounts. An operations lead may allow an agent to book standard service slots but escalate bookings involving scarce equipment. A finance lead may permit invoice retrieval but never let an external agent change bank details.
These are operating choices. IT can enforce them only after the business makes them explicit.
Run a one-hour route inventory
Do not begin with a large AI governance programme. Start with your website routes.
In one hour:
- List the pages, forms, APIs and portal actions that a machine can reach.
- Mark each one Discover, Retrieve or Transact.
- Name the business value and the data involved.
- Define the identity, rate limit and allowed action.
- Set the approval threshold, logging requirement and revocation method.
- Assign one accountable owner.
You will probably find routes that were designed for a person clicking slowly but are now reachable by software acting continuously. You may also find useful public data that is unnecessarily difficult for legitimate assistants to retrieve.
Both findings matter.
SEO asks, “Can machines find us?”
An agent-access policy asks the more complete question: “Once they find us, what are they allowed to do?”
Sources
[1] https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots — Cloudflare Block AI Bots documentation