Arrow 
Allow the documented search crawler for the engine you want to reach, then check delivery through your CDN. Make a separate decision about training. For OpenAI, OAI-SearchBot handles search, GPTBot concerns potential training, and ChatGPT-User handles certain user-initiated visits.
Check your current rules with the free AI website checker. It inspects public HTML and declared Googlebot and OAI-SearchBot access, with training preferences reported separately.
Which crawler controls which use?
| Provider / control | Purpose | Practical decision |
|---|---|---|
| OpenAI / OAI-SearchBot | ChatGPT search discovery | Allow public sources if search visibility is intended. |
| OpenAI / GPTBot | Potential foundation-model training | Decide separately from search. |
| OpenAI / ChatGPT-User | Certain user-initiated actions | Not a replacement for the search crawler. |
| Google / Googlebot | Google Search crawling | Review Search access and indexing controls. |
| Google / Google-Extended | Other documented Google AI uses | Review its scope separately from Search. |
| Other providers | Provider-specific search, training or user actions | Verify the current documentation before adding a rule. |
Start with OpenAI’s crawler reference and Google’s guidance for generative Search features. A company name or a generic “AI bot” label does not establish the request’s purpose.
How do you choose a policy?
Write down the public content you want discoverable, the training uses you permit and the routes that must remain restricted. Do this per provider and host. A provider may offer separate identities for separate uses; do not assume that every operator does.
For a specific OpenAI decision, read OAI-SearchBot vs GPTBot. To implement it, use the search-without-training configuration example, including repeated crawl exclusions for matching groups.
What should robots.txt accomplish?
It should express the intended crawl policy for that host. Keep public-page allowances, sensitive-path exclusions and provider-specific preferences consistent. Robots rules are voluntary crawl instructions, not authentication; private information must be protected by the application.
Before copying an example, check how matching groups are selected. A dedicated agent group can change which rules apply. Review the response at the public robots.txt URL after deployment, especially if a CDN adds managed directives.
Consult RFC 9309 for protocol semantics. An llms.txt directory serves a different purpose and does not replace these controls.
Why can the firewall still block a permitted bot?
The crawler may be allowed by robots.txt but receive a challenge, rate limit or access denial from the CDN. Inspect status, headers and the body of a public GET response, then compare the request with security events. Verify provider identity using supported methods or published IP ranges; a user-agent string can be spoofed.
The Cloudflare 403 and challenge guide shows how to find the responsible rule and verify a scoped correction. Do not infer successful article delivery from a 200 status alone.
What does allowing a crawler actually prove?
An allowance expresses a policy. A verified successful fetch demonstrates access for an observed request. Neither is an indexing report or a citation result. Keep those stages separate in your reporting.
For an illustrative documentation site, a valid release test checks public article delivery, the intended training preference and continued login requirements on account routes. Later, use recorded ChatGPT Search observations to check whether the page is selected as a source.
Sources and editorial scope
Reviewed October 21, 2026. This guide provides a decision process; it does not promise search inclusion. Provider policies and verification methods can change.