# Does Letting AI Read Your Site Mean Training? Search, Citation, and Training Are Different

- Canonical URL: https://oka-project.com/en/blog/ai-crawlers-search-vs-training
- Published: 2026-10-09
- Author: oka (https://oka-project.com/en/about)
- Summary: Showing up in AI search and being used for AI training are not the same thing. Using OpenAI, Google, and Anthropic documentation as of September 2026, I compare search, training, user-initiated fetching, and Gemini grounding.
- Category: research / Tags: AI, Web, robots.txt, search

---
When you read about AI search and generative AI, phrases like "letting AI read your site," "showing up in AI search," and "allowing AI training" are often used as if they meant the same thing.

But from the point of view of someone running a website, are they really the same?

You might want your pages to appear in ChatGPT search but not be used to train future models. Or you might want to stay in regular Google Search while limiting some uses by Gemini.

I wanted to know whether settings like that are possible, so I compared the official documentation from OpenAI, Google, and Anthropic as of September 25, 2026.

The short answer: **there is more than one kind of "AI reading your site."**

## The short version: search, training, and user-initiated fetching are controlled separately

Looking across the major AI services, it makes sense to treat at least these uses as separate:

| Use | Why it reads your pages | Examples |
| --- | --- | --- |
| Training crawls | Collecting web content that may be used to develop and improve future models | GPTBot, ClaudeBot |
| AI search and indexing crawls | Improving the relevance of current search results and answers | OAI-SearchBot, Claude-SearchBot |
| User-initiated fetching | Fetching a page on the spot to answer a user's question | Claude-User |
| Regular search crawls | Collecting pages for traditional web search | Googlebot |
| Generative AI features inside Google Search | Showing links or supporting content in AI Overviews, AI Mode, and similar features | Search Console's generative AI settings |

In other words, **appearing in AI search and being used to train future models are not the same switch.**

OpenAI separates `OAI-SearchBot`, used for search, from `GPTBot`, which controls use for future training. It also defines `ChatGPT-User`, which accesses pages in response to user actions. Anthropic publishes three separate bots: one that collects candidate training data, one for search, and one for user-initiated fetching. Google lets you control regular search, generative AI features inside Search, and generative AI model training through different mechanisms.

If you do not know about these differences and simply "block every AI bot" or "allow everything so AI can read my site," you may end up with something other than what you intended.

## Side by side, it is clearer what each control actually does

| Provider | Search and current answers | Future model development | User-initiated fetching | Main controls |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User | robots.txt / noindex |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User | robots.txt |
| Google | Googlebot + Search Console's generative AI settings | Uses covered by Google-Extended | Google's user-triggered fetchers | robots.txt / Search Console / noindex |

The key point of this table: **even though the names and mechanisms differ by company, search, training, and user-initiated fetching are not the same concept.**

Google in particular splits the controls for regular search, generative AI features inside Google Search, and training plus some grounding uses for Gemini models.

## OpenAI: ChatGPT search and future training can be handled separately

OpenAI's FAQ for publishers says that if you want your content included in ChatGPT search summaries and snippets, you should not block `OAI-SearchBot`.

For sites or pages you want excluded from future training, it says to disallow `GPTBot`.

Reference: [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)

Reference: [OpenAI: Publishers and Developers - FAQ](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq)

### You can allow OAI-SearchBot and block GPTBot

For example, this configuration:

```txt
User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /
```

expresses the policy "keep the possibility of being found and cited in ChatGPT search, but refuse GPTBot's crawling for future training."

This means **search visibility and opting out of training can be decided separately.**

### ChatGPT-User is not a search crawler

In OpenAI's current crawler documentation, `ChatGPT-User` is described as **the user agent used when ChatGPT or Custom GPTs access a web page because of a user's action.**

It is not a bot that crawls the web automatically to decide what appears in search, like OAI-SearchBot, and it is not used to manage inclusion in search. Because the access is user-initiated, OpenAI also says that robots.txt rules may not apply.

This means that blocking `OAI-SearchBot` does not necessarily stop ChatGPT from ever accessing your URLs.

It is more accurate to think of it this way: OAI-SearchBot is what you manage for search inclusion, GPTBot is how you express your wishes about training crawls, and ChatGPT-User is user-initiated fetching.

### robots.txt changes do not necessarily reach ChatGPT search immediately

OpenAI says that after you change robots.txt, it **can take about 24 hours** for ChatGPT search's systems to adjust.

So it is better not to check search results right after a change and conclude "I blocked it but it still shows up" or "I allowed it but it is not there yet."

### Blocking in robots.txt does not always make the URL disappear

OpenAI's FAQ has one more important caveat.

Even for a page OAI-SearchBot cannot crawl, if the URL is obtained from somewhere like a third-party search provider and there are relevance signals, ChatGPT Atlas may show just the link and the page title.

If you want a page completely kept out of search results, you need `noindex`. But a crawler has to be able to crawl the page to read that meta tag.

What this shows is that these are three different things:

- robots.txt: your crawling policy
- noindex: your indexing and search display policy
- authentication and access control: not making the page public in the first place

**robots.txt is not access control.**

### You can see ChatGPT search traffic through utm_source

OpenAI's current FAQ says that ChatGPT search automatically adds `utm_source=chatgpt.com` to URLs it sends people to.

So beyond deciding whether to allow OAI-SearchBot, you can check in Google Analytics or similar tools **whether traffic from ChatGPT is actually arriving** after you allow it.

If you are working on AI search visibility, measuring referral traffic matters more in practice than just having "allowed" the crawler.

## Anthropic splits "AI reading your site" into three kinds

Anthropic's official help makes this distinction even clearer.

When I rechecked Anthropic's current help on September 25, 2026, it still listed three bots:

| Bot | Main purpose |
| --- | --- |
| ClaudeBot | Collects web content that may be used in future training to make models more helpful and safe |
| Claude-User | Fetches web pages in response to Claude users' questions |
| Claude-SearchBot | Crawls the web to improve the relevance and accuracy of search results |

Reference: [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)

### ClaudeBot collects candidates for future training

Anthropic explains that restricting ClaudeBot signals that your site's future material should be excluded from AI model training datasets.

This is a role similar to OpenAI's GPTBot.

### Claude-SearchBot crawls for search quality

Claude-SearchBot analyzes online content to improve the relevance and accuracy of search results.

Anthropic says that blocking it prevents your content from being indexed for search optimization, which may reduce your visibility and accuracy in search results.

### Claude-User fetches a page because a user asked right now

The most interesting one is `Claude-User`.

It is used when a user asks Claude a question and Claude accesses a web page to answer it.

So the uses are split all the way down to:

- crawling regularly for future model development
- indexing for search quality
- fetching a page right now to answer this particular user's question

That gives site owners room for in-between positions, such as:

> I do not want my content used for training, but I do want it to be found through search.

> If a user passes my URL directly to the AI, I want it to be readable.

## Since late August 2026, Google separates AI in Search from AI training even further

Google works a little differently from OpenAI and Anthropic: regular search, generative AI features inside Search, and AI model training each have their own controls.

Starting August 31, 2026, Search Console's **generative AI settings for Search** rolled out to sites worldwide. There you can manage whether your site's links and content are included in or excluded from AI Overviews, AI Mode, and generative AI features in Google Discover.

Reference: [Search Console Help: Generative AI settings for Search](https://support.google.com/webmasters/answer/16908024?hl=en)

### Changes take time to apply here too

Google says that after you change the generative AI settings in Search Console, the change usually reaches the affected features within a few days, and that exclusion **often happens within one to two days** after the setting takes effect.

Caching and propagation across systems can make it take longer, though. Here too, it is safer not to judge success or failure from a single check right after the change.

### Opting out of AI in Search is not a regular ranking setting

Google explains that this setting is not used as a signal for ranking or inclusion in regular Google Search.

So as of September 2026, Search Console lets you choose to:

- appear in regular Google Search
- stay out of AI Overviews, AI Mode, and similar features

### Inclusion in AI search and control over model training are also separate

Search Console's generative AI settings do not affect Google's AI training. Google says that if you want to restrict training of the models that generate answers in Search's generative AI features, you should use `Google-Extended`.

But `Google-Extended` covers more than training. Google's current crawler documentation says it also controls **the use of the Google Search index to ground answers in Gemini Apps and Vertex AI**, in addition to training future Gemini models.

`Google-Extended` is not a separate HTTP user agent from the regular Googlebot. It is a product token used in robots.txt to control how content is used. It does not affect inclusion or ranking signals in Google Search.

So with Google, you need to separate at least the following:

| Goal | Main control |
| --- | --- |
| Appear or not appear in regular search | Googlebot / noindex, etc. |
| Be included in AI Overviews, AI Mode, etc. | Search Console's generative AI settings for Search |
| Restrict Gemini model training and some grounding uses in Gemini Apps / Vertex AI | Google-Extended |

This is the part that changed most around the time I wrote this. Previously, AI Overviews had to be explained mainly in terms of the regular search controls; now Search Console has a dedicated setting.

### You can also measure generative AI exposure separately in Search Console

Also starting August 31, 2026, Search Console's **generative AI performance report** rolled out worldwide.

It shows impressions where links to your site appeared in AI Overviews and AI Mode, broken down by page, country, date, device, and so on.

Reference: [Search Console Help: Generative AI performance report (Search)](https://support.google.com/webmasters/answer/16984139?hl=en)

Note that **this "Search" report covers AI Overviews and AI Mode. Generative AI features in Discover are covered by a separate generative AI performance report for Discover.** It is worth keeping the scope of the settings and the reports clearly apart.

Reference: [Search Console Help: Generative AI performance report (Discover)](https://support.google.com/webmasters/answer/16983858?hl=en)

That means you can now handle "should my site be included in AI search" (a setting) and "which pages actually appeared in AI search" (a measurement) separately inside Google Search Console, more clearly than before.

## Being searched, being trained on, and being read on the spot happen on different timelines

The differences become even clearer when you look at when the information is used.

### Training crawls serve future model development

Just because a page was crawled today does not mean its content shows up in answers today.

Training and updating a model is a separate process.

### Search and indexing crawls serve current questions with fresh information

AI search can draw on a search index or web search to refer to current information, not just what the model already knows.

That is a different mechanism from training.

### User-initiated fetching is closer to "read this URL now"

Access like Claude-User's is much more about fetching a web page at that moment to handle a user's request.

Describing all of this as "AI is training on it" erases these differences in timing too.

### Write down four goals before changing any settings

For robots.txt and similar settings, decide **what you want to achieve** first, then map it to each company's latest documentation. Crawler names and policies change, so do not treat the lists in this article as a permanent configuration spec.

| Site owner's goal | What to look up | What not to expect |
|---|---|---|
| Be found in regular search | The conditions for regular search crawling and indexing | A guarantee that writing Allow means you will be listed |
| Be featured or cited in AI search and answers | The paths the service uses for search and citation | A guarantee that allowing access means you will be cited |
| Handle use for future model training separately | The training controls each company publishes | A guarantee that information already distributed will disappear automatically |
| Protect non-public information | Authentication, scope of publication, appropriate delivery settings | The idea that robots.txt alone can keep something secret |

When you change these settings, save the previous robots.txt and meta tags, and record the date of the edit, the official documents you relied on, and the actual HTTP responses your site returns. **What a policy says, whether crawlers follow it, and what happens in search and citation are separate things to observe.**

## Decide your policy per goal, not "allow AI or not"

Rather than memorizing every bot name, it is easier to start by deciding what you want to allow on your own site.

### Do you want to appear in regular search?

This is your policy for traditional search engines like Google.

For a public site, you will usually want to allow it.

### Do you want to be cited or featured in AI search?

This is whether you see being discovered through ChatGPT, Claude, and similar services as a benefit.

For service sites and content sites, allowing it as a new discovery channel is a reasonable choice.

### Are you OK with possible use for future model training?

Decide this separately from search visibility.

"I want to appear in AI search but block training bots" is a perfectly valid policy.

### Should your pages be readable when a user gives the URL directly to an AI?

For public material, you may want to allow it; for paid or members-only content, you may decide differently.

Just deciding these four things makes the intent behind your robots.txt much clearer.

## robots.txt is not a way to keep things secret

Comparing what OpenAI, Anthropic, and Google say, robots.txt is the main way to communicate your wishes to each crawler.

But robots.txt is not authentication or access control.

It is dangerous to think that contracts, customer information, internal documents, or paid members-only content are safe "because I disallowed them in robots.txt."

Information you really do not want public needs other mechanisms: login, access control, proper authorization.

It is best to treat robots.txt as **a way to tell automated crawlers on the public web what your crawling policy is.**

## llms.txt does not replace training, search, or crawling controls

`llms.txt` also comes up as an AI-related web practice.

But it is better kept separate from the question this article is about: who is allowed to do what.

robots.txt tells specific crawlers whether they may access your pages.

llms.txt, on the other hand, is spreading as a proposal to help LLMs understand a site's content. It is not a standard for managing training opt-outs across OpenAI, Google, and Anthropic in one place.

So the relationship is not:

> I added llms.txt, so I have allowed AI training.

> If I do not write llms.txt, AI will not read my site.

As a real example, oka-project has an [llms.txt](https://oka-project.com/llms.txt) as an entry point that describes what is on the site, but permission for crawlers and training uses is managed separately, through robots.txt and each company's settings.

## In the AI era, robots.txt is becoming a statement of intent per use, not just a block list

robots.txt used to feel like a plain configuration file for telling search engines "please do not crawl here."

Looking at the major services in 2026, its role seems to have widened a little:

Search.

Future training.

Improving search quality.

User-initiated fetching.

Even within one company, access is split by purpose, and sites can choose.

What matters in the AI era is not a yes-or-no choice of whether to let AI read your site, but

**who gets to read it, for what purpose, and how far.**

I think deciding that is the real task.

As someone who runs a site myself, the most realistic position feels like the middle ground: I want to be in search, I want AI search to find me too, but there is no need to allow every use unconditionally.

At the very least, **appearing in AI search does not mean you have automatically allowed model training.**

### References

- [OpenAI: Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)
- [OpenAI: Publishers and Developers - FAQ](https://help.openai.com/en/articles/12627856-publishers-and-developers-faq)
- [Google: Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers?hl=en)
- [Search Console Help: Generative AI settings for Search](https://support.google.com/webmasters/answer/16908024?hl=en)
- [Search Console Help: Generative AI performance report (Search)](https://support.google.com/webmasters/answer/16984139?hl=en)
- [Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)