AI Visibility Academy · Chapter 08
Claude: how it searches and cites the web
Claude answers from what its model learned in training and, when a question needs current information, from a web search it runs itself. Anthropic's help center says that when Claude searches, every response includes citations so readers can check the sources. Web search has been available on all Claude plans worldwide since May 2025. To be one of those citations, your pages have to be reachable by Anthropic's crawlers and clear enough to quote.
When Claude goes to the web
Claude searches when a question benefits from current information. In Anthropic's newer interface there's no search toggle: Claude decides. If a user pastes a URL, Claude can fetch that page directly, and for local questions it may use the user's approximate location. Its Research feature runs multiple searches that build on each other before answering, with citations.
Three crawlers, three jobs
As of October 2026, Anthropic documents three bots:
| Bot | Job | If you block it |
|---|---|---|
| Claude-SearchBot | Indexes pages to improve search results | May reduce your visibility and accuracy in search answers |
| Claude-User | Fetches pages when a user's question needs them | Claude can't retrieve your page for that user |
| ClaudeBot | Collects content that may be used for training | Your future content should be left out of training |
Anthropic says its bots honor robots.txt, that rules must be set on each subdomain, and that blocking its IP addresses may not reliably opt you out, because it can stop the bots from reading robots.txt at all. If you want to be cited, allow the first two. The third is a separate choice, covered in search crawlers vs training crawlers.
What isn't documented
Anthropic's help pages don't say which search index Claude uses for web results, or how it chooses which results to cite. The help center does say image results are powered by Bing. Anthropic's published list of subprocessors names Brave Search among its web search providers, which tells you who handles search requests, not how sources are picked. So there's no single index to optimize for. Keep your pages fetchable, current and easy to quote.
This week
Open your robots.txt and look for Claude-SearchBot, Claude-User and ClaudeBot, and for a * rule that blocks them by default. Check that your CDN or firewall isn't blocking AI bots before they reach the page. Then ask Claude three buying questions with search on and note which pages it cites.
Sources
- Claude Help Center, "Enable and use web search", accessed October 8, 2026 (opens in a new tab)
- Anthropic, "Claude can now search the web", March 20, 2025, updated May 27, 2025 (opens in a new tab)
- Anthropic, "Does Anthropic crawl data from the web, and how can site owners block the crawler?", April 7, 2026 (opens in a new tab)
- Anthropic, "Claude takes research to new places", April 15, 2025 (opens in a new tab)
- Anthropic Trust Center, "Subprocessors", accessed October 8, 2026 (opens in a new tab)