Which AI crawlers should an online store allow, and how do you set that?
Allow the crawlers that fetch pages for answers, and decide separately on the ones that collect text for training. For a store that wants to be cited, that means allowing OAI-SearchBot, PerplexityBot, and Bingbot. GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended are the training controls, and they are not the same switch.
The agents, and what each one is for
OpenAI documents three agents that matter here, and the settings are independent. The page is https://developers.openai.com/api/docs/bots.
OAI-SearchBot is for search. OpenAI says it is used to surface sites in ChatGPT search. A site that disallows it is not shown in ChatGPT search answers, and it can still appear as a navigational link. OpenAI says a robots.txt change can take about 24 hours to be picked up. The published IP list is https://openai.com/searchbot.json.
GPTBot crawls content that may be used to train generative models. Disallowing GPTBot says the content should not be used for that training. It does not, by itself, disallow OAI-SearchBot. If you allow both, OpenAI says it may use one crawl for both purposes, so an opt-out of training has to name GPTBot. The IP list is https://openai.com/gptbot.json.
ChatGPT-User fetches a page because of a user action in ChatGPT or a custom GPT. OpenAI says it is not the automatic search crawler and not the training crawl. The IP list is https://openai.com/chatgpt-user.json.
OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user started the fetch. A Disallow for that token is not a reliable block. Search opt-out is OAI-SearchBot.
Perplexity documents two agents at https://docs.perplexity.ai/docs/resources/perplexity-crawlers. PerplexityBot surfaces and links sites in Perplexity search. Perplexity says it is not used to crawl content for foundation-model training. Perplexity-User fetches a page because a user asked, and Perplexity says that fetcher generally ignores robots.txt. IP lists: https://www.perplexity.ai/perplexitybot.json and https://www.perplexity.ai/perplexity-user.json.
Anthropic documents separate robots. ClaudeBot collects content that may contribute to model training. Restricting it signals that future material should stay out of training sets. The privacy page is https://privacy.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler. The same page describes Claude-SearchBot for search and Claude-User for a user-requested fetch. Blocking only ClaudeBot does not state a preference about Claude search.
Google-Extended is a robots.txt token, not a crawler you will see in the logs under that name. Google's crawler overview says it controls use for Gemini Apps and Vertex AI, and that it does not affect inclusion or ranking in Google Search. It has no HTTP user agent of its own. AI Overviews and AI Mode follow Googlebot. The AI-features page says a robots.txt rule for Googlebot controls crawling for Search, and it points to Google-Extended for other Google systems. Disallowing Google-Extended does not remove a page from AI Overviews. The pages are https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers and https://developers.google.com/search/docs/appearance/ai-features.
Bingbot is the crawler for Bing. Bing's crawler list does not name a separate Copilot crawler. The webmaster guidelines say to avoid blocking Bingbot, and that noindex keeps a URL out of Bing search, Copilot, and grounding results. noarchive keeps the page content out of Copilot answers. The guidelines are https://www2.bing.com/webmasters/help/webmaster-guidelines-30fba23a.
Applebot-Extended does not fetch pages. Apple says Applebot crawls, and Applebot-Extended only controls whether that crawled data may train Apple foundation models, including Apple Intelligence. A page can disallow Applebot-Extended and remain eligible for Apple search. The page is https://support.apple.com/en-us/119829.
What to open, and what to close
Open the product URLs, the category URLs you are willing to have quoted, and the policy pages. Allow OAI-SearchBot, PerplexityBot, and Bingbot if the goal is to be fetched for answers. Allow Googlebot if you want Google Search.
Close training only if that is the decision. A Disallow for GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended is that decision. It is not a substitute for an Allow on the search agents.
Close the cart, the checkout, the account, and filter URLs that multiply without end. Do not disallow product URLs if those are what you want cited. A firewall can still block an agent that robots.txt allows. Check that layer too.
A robots.txt example
Put this at https://example.com/robots.txt, on the host people type. Rules on the www host do not cover the bare host. The tokens are case-sensitive. GPTBot is not gptbot. OAI-SearchBot keeps the hyphens.
This example allows search fetches and disallows the training controls. Swap the training lines to Allow if you want those uses too. The star group still hides the cart.
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /accountWhat llms.txt is
llms.txt is a convention, not a robots.txt replacement. The proposal is at https://llmstxt.org/. A store puts a short Markdown file at /llms.txt. It names the site and links the pages worth reading. Crawlers are not required to fetch it. It does not grant or deny access. robots.txt does that.
Keep it short. Link the catalog, shipping, and a page that states how prices work. Do not paste the catalog into the file.
# Example Store
> Outdoor goods sold from Warsaw. Prices on product pages are in PLN and include VAT.
## Pages
- [Catalog](https://example.com/collections/all): Current products, with price and stock on each product URL.
- [Shipping and returns](https://example.com/policies/shipping): Delivery countries and the return window.
- [Contact](https://example.com/pages/contact): How to reach the shop.How to see them in the logs
Search the access log for the user-agent token, not for a marketing name. The tokens are OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Claude-SearchBot, Claude-User, Bingbot, and Applebot.
Do not wait for a line that says Google-Extended or Applebot-Extended. Those are robots.txt tokens. Googlebot and Applebot are the clients that fetch.
Match an OpenAI request against the published JSON IP lists linked above before you treat a spoofed user-agent as real. A client can send any user-agent string.
Crawler prefixes are at https://claude.com/crawling/bots.json, one list for every bot. The privacy article says an IP on that list is Anthropic, and that blocking those IPs can stop the bots from reading robots.txt. Use the robots.txt token.
Keep the token, the path, and the status. OAI-SearchBot on a product URL with 200 is a search fetch. ChatGPT-User or Perplexity-User on that path is a user-triggered fetch. A token with no path does not show that the product URL was fetched.
Server HTML and client-rendered price
robots.txt only decides whether a fetch is allowed. It does not put the price into the answer. The product URL has to return the name, the price, and the stock status in the first HTML.
Check with curl, or with view-source, and search for the price. If the number appears only after hydration, the raw response does not contain it.
Google's JavaScript SEO guide says Googlebot renders JavaScript. That is one crawler. It is not a claim about the others.
The OpenAI, Perplexity, and Anthropic pages above do not say those agents run JavaScript. The server HTML is the copy they can read.
If the price is printed only in a client component, move it into the server render. curl the URL and search for the same digits.
- OAI-SearchBot, PerplexityBot, and Bingbot are allowed if the store wants to be fetched for answers.
- GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended are set on purpose, not copied from a block-all list.
- Cart, checkout, and account are disallowed without disallowing product URLs.
- The file is on the host shoppers use, and the WAF does not block the allowed agents.
- /llms.txt is a short map of pages. It is not the catalog and not an access control.
- A product URL returns the price in the first HTML response.
- Logs are searched for the user-agent token, then checked against the published IP list.
If I block GPTBot, do I disappear from ChatGPT search?
OpenAI says the controls are independent. Disallowing GPTBot is the training opt-out. ChatGPT search uses OAI-SearchBot. Leave OAI-SearchBot allowed if you want to stay eligible for those answers. If both are allowed, OpenAI may reuse one crawl for both, so the training opt-out has to name GPTBot.
Does Google-Extended control AI Overviews?
Do not treat it as that switch. Google-Extended is the token for the uses Google documents, separate from ordinary Googlebot crawling. AI Overviews follow Googlebot. Disallowing Google-Extended does not remove the store from AI Overviews.
Will an assistant skip a page that llms.txt does not list?
No. llms.txt is an optional map. robots.txt and the HTTP response decide access. A product page can be fetched and cited with no line in llms.txt.
Should I allow ChatGPT-User?
ChatGPT-User is the fetch a user action can trigger. It is not the search crawler. OpenAI says robots.txt rules may not apply, because the user started the fetch. Allowing the token does not put the product into an answer.