Short answer

Match a failed public-page request to its response body, timestamp, client identity and Cloudflare security event. Then change the specific policy responsible. An Allow in robots.txt cannot override an edge block, and a copied bot user-agent cannot prove a real crawler was allowed.

Capture the failing request before editing rules

Choose one public URL that should be accessible. Record the hostname, UTC time, response status, headers, final URL and a short body sample. Compare the same evidence after a targeted change. Keep account and administrative routes outside the public-page test.

How do you inspect what the crawler receives?

Use a GET request so you can inspect the content, not only HEAD headers. The following commands are a local diagnostic for a site you operate. Replace the example URL; do not paste cookies, tokens or private response bodies into a public report.

curl -sS -L --max-time 30 -D headers.txt \
  -o body.html https://example.com/public-guide/
curl -sS -L --max-time 30 -D robots-headers.txt \
  -o robots.txt https://example.com/robots.txt

Review every response in a redirect chain and inspect the saved body. If you repeat the request with a provider user-agent, label it as a simulation. It still comes from your network and can legitimately differ from the verified crawler’s request.

What do 403s and challenge headers tell you?

ObservationInterpretationNext check
403 with a corresponding security eventAn edge rule took action on this request.Read the action, service and rule identifier.
cf-mitigated: challengeCloudflare returned a Challenge Page.Inspect the responsible product and identity classification.
403 without an obvious matching eventThe origin or another layer may be involved.Compare origin logs; check event coverage and timing.
200 containing login, consent or error HTMLThe expected public article was not delivered.Inspect the body and application or cache behavior.
Expected article HTMLDelivery passed for this observed request.Verify real crawler identity and later source selection.

Cloudflare documents cf-mitigated: challenge as a Challenge Page signal. A Cloudflare response header alone does not prove Cloudflare caused an origin-generated 403; use its 403 troubleshooting guide with your logs.

How do you locate the responsible rule?

In Security Events, filter around the recorded time and hostname, then narrow by path, action, client IP and Ray ID where available. Retain the product or service, rule ID and the classification shown. Available fields, sampling and retention depend on your account; an empty view is not conclusive proof that no block occurred.

Inspect the current AI bot policies as well as custom rules, managed rules and rate limits. Cloudflare distinguishes Search, Agent and Training behaviors, including mixed-purpose crawlers. An AI policy set to Allow does not mean other security layers stop evaluating the request.

Check the current AI bot policy documentation rather than relying on an older screenshot of a single “Block AI bots” toggle.

What is an appropriate scoped fix?

First verify that the request belongs to the provider using supported bot verification or current published IP ranges. A user-agent substring alone is not identity proof. Write down the public hostname and paths that need access, and the exact security control that produced the false positive.

Adjust that policy or use a narrowly scoped supported exception with logging. Review Cloudflare’s Skip options before deployment: a Skip does not bypass every product, and Bot Fight Mode cannot be skipped through the same option as Super Bot Fight Mode. Preserve unrelated protections and a rollback path.

What does a successful retest prove?

Repeat the GET test and inspect subsequent verified crawler requests where available. Confirm that the intended article is returned, challenge signals are absent for that request, and protected routes still require authorization. Record the time, policy change and observed outcome.

This proves a bounded delivery result. Use the citation-tracking procedure to measure selection afterward. If the problem was a robots group rather than the firewall, use the search and training policy example.

Sources and editorial scope

Reviewed October 21, 2026 against the linked provider documentation. Procedures and examples are Arrow AI implementation guidance, not measured customer results. Policies and dashboard controls can change. Access does not guarantee indexing or citation.

The Arrow read

Make public sources reachable and verifiable.

Start with the crawler decision guide or run a free GEO audit to identify the next page worth improving.