Key takeaways
- robots.txt asks crawlers not to fetch pages. Reputable AI companies say they honour it, but it is voluntary.
- Training and search crawlers often have different names: block one without the other if you want AI search visibility.
- Google-Extended controls use of your content for Gemini training without affecting Google Search.
- Robots rules only affect future crawling, not data already collected.
On this page
How it works
robots.txt is a plain text file at the root of your site that lists which crawlers may fetch which paths, following the Robots Exclusion Protocol (RFC 9309). Each crawler identifies itself with a user-agent token. A group starting with User-agent: GPTBot followed by Disallow: / asks that crawler to stay away from the whole site.
Training vs search crawlers
Several companies now use separate crawlers for model training and for live search answers:
| Company | Training | Search / answers |
|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot (and ChatGPT-User for user-requested visits) |
| Anthropic | ClaudeBot | Claude-SearchBot (and Claude-User) |
| Google-Extended (a control token, not a separate crawler) | Googlebot (Search, including AI Overviews) | |
| Apple | Applebot-Extended | Applebot |
Company documentation changes, so check each provider's current crawler list before relying on a token.
Example: block training, allow AI search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlBuild your own with the robots.txt generator, then confirm each crawler's result in the robots.txt tester. A crawler obeys only the most specific group that names it, so a GPTBot group means GPTBot ignores the * rules.
What robots.txt cannot do
- It is a request. Crawlers that ignore it can only be stopped by server-side blocking (by IP or user agent at your host or CDN).
- It does not remove content already collected.
- It does not hide pages: the file itself is public, and blocked URLs can still appear in search if linked elsewhere.
- Blocking Googlebot to avoid AI Overviews also removes you from Google Search entirely.
Sources
- RFC 9309: Robots Exclusion Protocolrfc-editor.org
- OpenAI crawlersdevelopers.openai.com
- Google: common crawlers including Google-Extendeddevelopers.google.com
Fact-checked against the sources above. We test every tool we mention. Editorial policy.
