If your pages show up in ordinary search results but an AI assistant cannot open them, check crawler access at the network edge before you change anything on the site itself. Cloudflare’s AI Crawl Control lets a site owner allow or block individual AI crawlers at Cloudflare’s edge, and Cloudflare now enables AI crawler blocking by default on new zones added to it. That means a site can be refusing AI crawlers without its owner ever choosing that setting. The first move is reading your own Cloudflare dashboard, not rewriting your content.
What does AI crawler blocking at the edge actually do?
AI Crawl Control is a setting inside Cloudflare that decides, crawler by crawler, whether an AI company’s bot is allowed through. The decision is enforced at Cloudflare’s edge, which means the request is stopped before it reaches the origin server where your site lives. Your pages are not deleted, broken, or penalized. They are simply never handed over to that particular bot.
The part that surprises owners is the default. Because blocking is on by default for new zones added to AI Crawl Control, a business can move its DNS to Cloudflare for speed or security reasons and end up refusing AI crawlers as a side effect of that move. Nobody typed a rule. The setting arrived with the zone.
Why can’t my robots.txt file fix this?
A permissive robots.txt hosted on your site cannot override an edge block, and the reason is ordering. Google’s crawling documentation describes robots.txt as a file that lives at the root of a site and is fetched by the crawler over the network before it requests other URLs. A crawler has to be allowed to connect in order to read it.
If Cloudflare refuses the connection at the edge, the AI crawler never gets far enough to see your rules. You can write “allow everything” in that file, publish it, and it changes nothing for a bot that was turned away one layer earlier. This is why site owners who audit only their own files come away convinced everything is configured correctly.
Why do I still show up in Google but not in AI answers?
Classic search crawling and AI answer-engine crawling are done by different user agents. Google publishes a list of the crawlers it operates and what each one is for, and it introduced the separate Google-Extended control in 2023 so publishers could make a different choice for AI training and grounding than they make for Search. Other AI engines run their own named agents in the same way.
Because the agents are separate, a block can hit one group and miss the other. Googlebot keeps crawling, your pages stay indexed, and your ordinary listings look normal. Meanwhile the crawlers behind AI answer engines are refused. The symptom is invisibility in AI answers, not a fall in ordinary search listings, which is why this problem usually gets diagnosed as a content problem for months before anyone looks at the network layer.
What are the checkable signs that this is happening to me?
There are two signs, and you can look for both in about ten minutes.
Sign one: the page ranks but will not open
Find a published URL that appears in an ordinary search listing. Then ask an AI assistant to open and quote that exact URL. If the assistant returns an access error, or describes the page only from third-party summaries such as a directory listing or a news mention rather than quoting your actual wording, the crawler behind that assistant is not reaching your page.
Sign two: the dashboard says so
Open your Cloudflare dashboard, select the zone for your domain, and read the AI Crawl Control section. It shows which AI crawlers are set to block and which are set to allow. That screen is the record. If crawlers are set to block there, you have your answer without any further guesswork.
As a control test on the search side, Search Console’s URL Inspection tool will tell you whether Google can fetch a live URL. Keep in mind that it tests Google’s crawler, not an AI engine’s crawler, so a clean result there does not clear the AI side. It is useful mainly for confirming that the page itself is healthy.
If I allow the crawlers, will AI engines start citing my page?
No, and it is worth keeping the two things apart. A block is a refused connection. Whether an engine chooses to cite an accessible page is a separate editorial decision made afterwards, by that engine, based on its own judgment of the page.
So allowing a crawler removes an obstacle. It does not buy an outcome, and anyone who tells you otherwise is selling something. What you get is the ability to be considered at all, which is not the same as being chosen. We do not make guarantees about placement in AI answers, and this article covers crawler access only.
What should I do next?
Read your zone’s AI Crawl Control settings and write down which crawlers are blocked. Run the URL test above on two or three pages you would want quoted, such as a service page and a location page. If the dashboard shows blocks you did not choose, decide deliberately which crawlers you want through, because some owners genuinely do want to refuse some of them.
If you would rather not do this yourself, that is what we handle. Nauvativ automates website SEO and AEO for local small businesses, and a person approves every change before it ships. You can see how the work is scoped on our services page, or get in touch and we will look at your zone and your pages together.
Ask us to check your crawler access