AI Crawler Access: Robots.txt, Sitemaps, and Indexing Explained

For a page to show up as a sourced link in a web-grounded AI answer, the system generally needs a way to discover and fetch it. That doesn't mean every crawlable page gets indexed, retrieved, or cited. Discovery, crawling, indexing, retrieval, and citation are separate stages, and each one fails differently, so each one needs its own diagnostic.

What are the stages between publishing and citation?

The path looks like this:

1. Discovery. A crawler learns the URL exists through links, a sitemap, a feed, or a submission mechanism.
2. Crawl permission and access. Robots rules, authentication, firewall settings, and server responses either allow the fetch or block it.
3. Processing or indexing. A search system evaluates the fetched page and may add it to an index.
4. Retrieval. The system picks the page as relevant to a query or a generated subquery.
5. Use in an answer. The system may link, quote, summarize, or ignore what it retrieved.

A sitemap helps discovery but can't guarantee indexing. Allowing a crawler makes access possible but doesn't guarantee retrieval. And a citation only proves a link appeared in one particular answer, not that the page holds some permanent "AI ranking." What happens at the retrieval stage is a topic of its own; we dug into it in how AI engines choose citations.

Google's documentation says a page must be indexed and eligible to appear in Search with a snippet to qualify as a supporting link in AI Overviews or AI Mode, and it stresses that crawling, indexing, and serving are not guaranteed.

What should robots.txt do for AI search?

robots.txt tells compliant crawlers which paths they may fetch. Review rules by user agent; don't assume every bot obeys the same line.

OpenAI documents separate purposes for its crawlers. OAI-SearchBot is used to surface sites in ChatGPT search features, while GPTBot relates to potential model training. OpenAI says the settings are independent: you can allow OAI-SearchBot and disallow GPTBot at the same time.

Before you edit robots.txt:

• Decide which experience you want to enable or restrict
• Read the crawler owner's current documentation, not a cached summary
• Inspect the specific path rules and wildcard behavior
• Check CDN, web application firewall, rate-limit, and bot-management settings too
• Confirm the crawler actually receives the intended status and content
• Keep restrictions on private, account, checkout, search-result, and other sensitive or low-value areas

Don't copy a universal allowlist from an undated blog post. User agents, IP ranges, and platform policies change.

What does an XML sitemap contribute?

An XML sitemap lists the canonical URLs you want search systems to find. It earns its keep when it's accurate, current, and limited to URLs that return successful responses and are meant to be indexed.

A healthy content sitemap:

• Uses absolute canonical URLs
• Excludes redirects, error pages, duplicate parameter variants, and private pages
• Updates lastmod when the page materially changes, not on every deploy
• Stays within the protocol's size limits
• Is referenced from robots.txt where appropriate
• Gets submitted through supported webmaster tools

Google's sitemap documentation is blunt about this: a sitemap is a hint. It helps discovery; it doesn't guarantee crawling or indexing.

Is llms.txt required for AI visibility?

No. There's no universal standard that makes llms.txt a requirement for AI-search inclusion. It can serve as a concise, machine-readable map for tools that choose to read it, but the effect depends entirely on the consumer.

Google is explicit that it does not use llms.txt for Google Search, including generative AI features, and says the file neither helps nor harms Google Search visibility. Any claim that "an llms.txt file makes you rank in AI" is unsupported unless a specific system documents that behavior.

A maintained resource map isn't useless, though. It's cheap to serve, and some tools do consume it. CiteCue's AI Auto-Fix generates and serves an llms.txt for your site on every plan, alongside AI-optimized page variants. Just be honest about what the file does: your visible site still needs crawlable, indexable, helpful pages.

How do you diagnose an access problem?

Test from the outside in:

1. URL response. Does the canonical URL return a successful status without a login, region block, or challenge?
2. Rendered content. Is the important information present in readable text after rendering?
3. Robots policy. Is the relevant crawler allowed on the exact path?
4. Infrastructure. Is the CDN or firewall blocking the request or serving a challenge?
5. Discovery. Is the URL internally linked and present in a clean sitemap?
6. Index status. What do the relevant webmaster tools report?
7. Observed retrieval. Does the page show up for the questions you track? If not, which pages do? If you monitor prompts in CiteCue, the recorded citations show which URLs each engine actually used, which answers this quickly.

Keep server logs and referral analytics separate. A bot request is crawler activity; a human arriving from an AI answer is referral traffic. There's a short walkthrough on measuring AI referral traffic in Google Analytics if you haven't set that up yet.

Move from technical eligibility to useful evidence

Once access and indexing are healthy, stop treating the crawl layer as the whole strategy. Technical eligibility removes a blocker; it doesn't make your page the best available source. That part comes down to the quality of the answer, the evidence behind it, and whether the facts are still current.

CiteCue's AI Readiness runs crawler access checks, sitemap coverage, and connected Search Console index status in one place, and AI Referrals uses your connected Google Analytics to show human traffic arriving from AI chat surfaces. Put those next to your prompt and citation results and you can tell "the page can't be reached" apart from "the page was reachable but another source answered better."