aideagoCan AI bots read your website? A robots.txt checklist
Before an AI assistant can mention you in an answer, it has to be able to read your site. The programs that read websites are called bots (crawlers), and a small text file at the root of your site tells them who's allowed in: robots.txt. One wrong line in that file can remove your site from ChatGPT's search answers entirely.
Look at your file (1 minute)
Type your site address into the browser and add /robots.txt to the end. Example: examplehotel.com/robots.txt. You'll see one of three things:
- No file: no barrier for bots; that's fine.
Disallow: /underUser-agent: *: the whole site is closed to every bot. It may have been left over from when the site was being built; fix it right away.- Lines naming bots one by one: compare them with the list below.
Which bot does what?
According to the companies' own documentation, bots fall into two groups: those that read for search and answers, and those that collect content for model training. You can manage each separately.
For search and answers (keep these open if you want to be found):
- Googlebot: Google Search's bot. Google's AI overviews and AI Mode use the normal search index too; block Googlebot and you drop out of both.
- OAI-SearchBot: for ChatGPT's search feature. OpenAI: sites that block this bot won't be shown in ChatGPT's search answers. It can take about 24 hours for a change to take effect on OpenAI's side.
- PerplexityBot: to show and link websites in Perplexity's search results. According to Perplexity, it isn't used to train models.
- Claude-SearchBot: reads content to improve the quality of search results for Anthropic's Claude. If blocked, your visibility in user searches may drop.
Pages opened at a user's request:
- ChatGPT-User, Perplexity-User, Claude-User: step in when a user has a chat open your page. OpenAI and Perplexity note that because these visits are user-initiated, robots.txt rules may not apply.
For model training (your choice):
- GPTBot (OpenAI), ClaudeBot (Anthropic): collect content for training AI models.
- Google-Extended: not a separate bot but a token used in robots.txt. It controls whether your content is used for training Google's AI models and in some of its other systems. According to Google, Google-Extended doesn't affect a site's inclusion or ranking in Google Search.
In short: closing training bots doesn't remove you from search answers; closing search bots does.
A sample robots.txt
A simple example for a business that wants to be found in search and AI answers but keep its admin area closed:
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://examplehotel.com/sitemap.xmlIf you'd like to close the training bots, you can add these lines:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /robots.txt isn't the only door
Even with a correct file, bots can get stuck elsewhere:
- Hosting or firewall: some hosting and CDN (a network that speeds up your site) services have a "block AI bots" option. Google also recommends allowing crawling not only in robots.txt but in your CDN and hosting infrastructure. Check the bot settings in your panel.
- noindex tag: if a page's code says "don't index", Google won't show it in search.
- Text inside images: if your menu or price list is only a photo, bots may not be able to read it as text.
- Login walls: bots can't see pages that require an account.
10-minute checklist
- robots.txt opened and read.
- No
Disallow: /underUser-agent: *. - Googlebot, OAI-SearchBot, PerplexityBot and Claude-SearchBot aren't blocked.
- A conscious decision made about training bots (GPTBot, ClaudeBot, Google-Extended).
- The sitemap address is in the file.
- The "block AI bots" setting in your hosting panel checked.
- No noindex on important pages.
- Address, services and prices written as text.
After these checks, run the 15-minute ChatGPT test to see whether assistants actually mention you.