AI crawler access means deciding, bot by bot in your robots.txt file, which AI companies may read your site. The decision splits in two: bots that fetch pages so an AI search tool can cite you, and bots that collect pages for model training. Mixing them up is the most common and most expensive mistake, because blocking the wrong one removes you from AI answers.
What are the AI crawlers, and what does each one do?
The big vendors now publish separate bots for separate jobs. Here is what their own documentation says:
| Vendor | Bot (robots.txt token) | Job | Block it and… |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search | Your site is not shown in ChatGPT search answers (navigational links can remain) |
| OpenAI | GPTBot | Collects content that may train OpenAI models | Signals your content is not for training |
| OpenAI | ChatGPT-User | Visits a page when a user asks ChatGPT or a GPT to | robots.txt may not apply to user-initiated visits |
| Anthropic | Claude-SearchBot | Indexes content for Claude’s search answers | Visibility and accuracy in those answers may drop |
| Anthropic | Claude-User | Fetches pages for a user’s question | Stops retrieval for those questions |
| Anthropic | ClaudeBot | Collects content that could train Claude models | Signals future content is excluded from training |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results (not for training, per Perplexity) | You are not surfaced in results |
| Perplexity | Perplexity-User | Fetches pages for a user’s question | Generally ignores robots.txt |
| Google-Extended | A token controlling use for Gemini training and grounding outside Search | Does not affect Search inclusion, and is not a ranking signal | |
| Apple | Applebot-Extended | Controls use of Applebot data for Apple’s generative AI training. It does not crawl | Does not remove you from Apple search |
| Common Crawl | CCBot | Builds an open web archive that others use | Keeps you out of the archive |
Apple’s search crawler Applebot is separate: allowing it lets your pages appear in Spotlight, Siri and Safari results. Google’s AI Overviews and AI Mode use Googlebot and the normal index, so they follow your regular Googlebot and nosnippet rules (see AI Overviews and AI Mode).
Why does AI crawler access matter?
Three things depend on it. Visibility: a search bot that is blocked cannot cite you. Control: training bots decide whether your writing feeds future models. Server load and cost: each bot makes requests, and Anthropic supports the non-standard Crawl-delay rule to slow ClaudeBot.
There is no single right answer. A business that wants leads from AI answers allows the search bots. A publisher that sells its text may block the training bots too. Both are valid.
How do you set up robots.txt for AI bots, step by step?
- Decide your goal. Pick one of three: be cited and allow training, be cited and refuse training, or block AI altogether.
- Open your robots.txt at
yoursite.com/robots.txtand list everyUser-agentblock. Look for aUser-agent: *rule, because it applies to every bot that has no block of its own. - Add a block per bot. For “be cited, refuse training”, use:
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /- Repeat for each subdomain. Anthropic says to add the rule to the top-level robots.txt of every subdomain you want covered.
- Check the firewall. OpenAI and Perplexity advise allowing their published IP ranges (
openai.com/searchbot.json,perplexity.com/perplexitybot.json). Anthropic lists its ranges atclaude.com/crawling/bots.json. A CDN bot-protection rule that returns 403 overrides your robots.txt. - Do not block by IP to opt out. Anthropic warns that blocking IPs stops the bot from reading robots.txt and may not hold.
- Wait. OpenAI and Perplexity say changes can take about 24 hours.
- Test it. Request
robots.txtin a browser, then check your logs for each bot name and the status codes it receives.
For the syntax details, see robots.txt and noindex.
How do you measure AI crawler access?
Your server logs are the only source of truth. Count requests per bot name per week and watch the status codes: 200 means the bot reads your pages, 403 or 429 means your firewall is the problem. Verify suspicious traffic against the vendors’ IP lists, since anyone can fake a user-agent string (Common Crawl warns that other crawlers pretend to be CCBot).
For results, use the visibility checks on the ChatGPT search and Perplexity citations pages, and the Bing AI Performance report described on the Bing and Copilot page.
What mistakes should you avoid?
- Blocking Google-Extended and expecting it to hide you from AI Overviews. Google says it does not affect Search.
- Blocking everything with
User-agent: *andDisallow: /for “AI protection”. That also removes search engines. - Treating GPTBot and OAI-SearchBot as one bot. They are separate settings.
- Assuming robots.txt stops user-triggered fetches. OpenAI and Perplexity both say those visits may ignore it. Real protection needs authentication or a firewall rule.
- Adding
llms.txtand calling it done. Google says it does not use the file (see llms.txt). - Copying a 2024 blocklist. Bot names and roles change, so reread the vendor pages twice a year.
Where can you learn more?
Start with the vendor pages in the sources above. The free digital marketing module of the Fe/male Switch startup school shows how robots.txt fits into an SEO setup for a new site. Next steps: set up ChatGPT search access, then compare notes on Perplexity.