AI Visibility Academy · Chapter 03
Search crawlers vs training crawlers
A search crawler fetches your pages so an assistant can find, quote and link them in its answers now. A training crawler collects pages that may be used to train future models. Block a search crawler and you can drop out of that assistant's search answers. Block a training crawler and you don't. They are two decisions, and you can make them differently.
What each block does
As of October 2026, the owners describe the effect like this:
| If you block | What the owner says happens |
|---|---|
| OAI-SearchBot | Your site isn't shown in ChatGPT search answers |
| GPTBot | Your content shouldn't be used to train OpenAI's models |
| Claude-SearchBot | Anthropic can't index your content for search, which may reduce your visibility in Claude's search results |
| ClaudeBot | Your future content should be left out of training |
OpenAI says the two settings are independent, so allowing OAI-SearchBot while blocking GPTBot is a supported combination. Note the word "future" in Anthropic's description: a training block covers what gets collected from now on.
Google and Apple split it differently
Google uses one crawler, Googlebot, for Search, including AI Overviews and AI Mode. The two decisions are made in two different places:
- Training. The Google-Extended token in robots.txt limits use of your pages for training Gemini models and for grounding in the Gemini app and Vertex AI. Google says it doesn't affect inclusion or ranking in Google Search.
- AI search features. The Search generative AI control in Search Console, rolled out to all sites on August 31, 2026, can exclude your site from AI Overviews, AI Mode and generative features in Discover. It is set to include by default, and Google says it doesn't affect training.
Apple works like the Google-Extended half: disallowing Applebot-Extended limits training use, and Apple says those pages can still appear in its search results.
How to decide
For search crawlers the question is simple: if you want to be cited and linked in an assistant's answers, let its search crawler in. Blocking it is, in effect, opting out of that assistant's search.
Training is a real business choice. A publisher that licenses its content has reasons to block. A company that wants assistants to know its name and category even when they don't search may prefer to allow it, as covered in trained knowledge vs live search. How much training data shapes what a model says about one brand isn't documented by any of these companies, so treat it as a judgment call rather than a lever.
Where to start
Make a short grid: one row per assistant, one column for search and one for training, and write yes or no in each cell. Then check your robots.txt and your Search Console setting against it. Turning the grid into rules is the next lesson, writing robots.txt rules for AI crawlers.
Sources
- OpenAI, "Overview of OpenAI Crawlers", accessed Oct 8, 2026 (opens in a new tab)
- Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", Apr 7, 2026 (opens in a new tab)
- Google, "Google's common crawlers", Jul 14, 2026 (opens in a new tab)
- Google, "Search generative AI control", Search Console Help, accessed Oct 8, 2026 (opens in a new tab)
- Apple, "About Applebot", Sep 4, 2026 (opens in a new tab)