Short answer

Separate search and training policies by documented crawler identity. For OpenAI, you can allow OAI-SearchBot and disallow GPTBot. Repeat the crawl restrictions needed in each matching group, and check the robots.txt response served by your CDN.

Start with one provider and one host

This worked example targets OpenAI’s distinct search and training agents. It is not a universal “block AI training” switch. Before editing, inventory every hostname and provider your policy covers, including documentation and answer subdomains.

What should the robots.txt file contain?

The example below permits public-page search crawling, declines GPTBot collection and retains two illustrative crawl exclusions. Replace the hostname and paths. Apply it to the root robots.txt on the host you are changing, not inside an article directory.

# Example only: replace paths with your actual crawl restrictions.
User-agent: *
Disallow: /admin/
Disallow: /account/

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /account/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

The independent OpenAI controls are documented in OpenAI’s bot reference. The Robots Exclusion Protocol describes matching groups: a specific matching group does not simply inherit the wildcard group’s rules. Keep necessary exclusions in the relevant group.

What does this policy leave unchanged?

It does not control every provider, stop a noncompliant scraper, erase material already collected, or replace authentication. It also does not make llms.txt a consent mechanism. Treat confidential content as an access-control problem, with authenticated routes and appropriate authorization.

For other providers, verify whether search and training share an agent before copying the OpenAI pattern. If purposes cannot be separated through the documented control, record the trade-off rather than promising both outcomes.

Where do subdomains and CDN policies matter?

Check each intended scheme, hostname and port. A policy on example.com is not automatically the file served from docs.example.com. Fetch the public response after deployment and compare it with the version in your repository.

Cloudflare can prepend managed directives to an origin robots.txt. Review the combined response and the configured Search, Agent and Training policies, rather than assuming that the source file is the delivered policy. See Cloudflare’s robots.txt setting.

How should you deploy and verify the change?

  1. Save the previous file and write the intended search, training and private-path outcomes.
  2. Validate group matching for a public page and every excluded path.
  3. Deploy the file and fetch the public robots.txt with GET on each host.
  4. Inspect whether the public article reaches the intended HTML through the CDN.
  5. Observe verified crawler requests, then review citations separately. Keep the before-and-after records.

A useful acceptance test is “public documentation allowed; account crawl excluded; account access still requires login.” Neither a robots validator nor an anonymous browser visit proves every layer passed.

What if search access still fails?

Use the Cloudflare diagnostic for edge blocks and challenges. For an unexpected agent, check the OAI-SearchBot and GPTBot comparison before changing rules. Do not solve a public-page fetch problem by exempting an entire site from security checks.

When the access tests pass, keep the policy stable and measure comparable search observations. The configuration enables a documented access path; it does not purchase or guarantee visibility.

Sources and editorial scope

Reviewed October 21, 2026 against the linked provider documentation. Procedures and examples are Arrow AI implementation guidance, not measured customer results. Policies and dashboard controls can change. Access does not guarantee indexing or citation.

The Arrow read

Make public sources reachable and verifiable.

Start with the crawler decision guide or run a free GEO audit to identify the next page worth improving.