Step-by-step guide Get Recommended by ChatGPT: local service businesses →
Type your web address followed by /robots.txt into a browser. If the file has a group for OAI-SearchBot or for * followed by Disallow: /, your site has asked ChatGPT's search crawler to stay out, and OpenAI says such sites "will not be shown in ChatGPT search answers". Two things trip people up. The crawler that matters for ChatGPT search is OAI-SearchBot, not the better-known GPTBot. And a clean robots.txt doesn't prove the door is open, because a host setting or a firewall can still turn crawlers away. Here are four checks, in order, with our own site as the example.
Three OpenAI agents, three different jobs
OpenAI's overview of its crawlers, as it read on 10 November 2026, lists these three for website owners:
| Name | What OpenAI says it is for | What blocking it does |
|---|---|---|
| OAI-SearchBot | Surfacing websites in ChatGPT's search features | Your pages are left out of ChatGPT search answers. The site can still appear as a plain link |
| GPTBot | Crawling content that may be used to train OpenAI's models | Signals that your content should not be used for training. No effect on search |
| ChatGPT-User | Fetching a page when a person's question, or a custom GPT, calls for it | Uncertain. OpenAI says robots.txt rules may not apply, because a person started the action |
OpenAI states that each setting is independent of the others. So blocking GPTBot doesn't remove you from ChatGPT search, and allowing GPTBot doesn't put you in. A "block AI" snippet copied from a blog post can shut out both without the owner realising. A fourth agent, OAI-AdsBot, only visits pages submitted as ads.
The same split at Google, Perplexity and Anthropic
The same split between search and training shows up at the other large AI companies. The names to look for:
| Company | Feeds search answers | Controls training |
|---|---|---|
| OpenAI (ChatGPT) | OAI-SearchBot | GPTBot |
| Google (AI Overviews, AI Mode) | Googlebot | Google-Extended |
| Perplexity | PerplexityBot | None listed |
| Anthropic (Claude) | Claude-SearchBot | ClaudeBot |
Google's guidance on AI features says there are no additional requirements to appear in AI Overviews or AI Mode: a page that Googlebot can crawl and Google can show in ordinary search is eligible. Google-Extended is a separate switch for Gemini training, and Google's crawler documentation says it does not affect a site's inclusion in Google Search. Perplexity's crawler page recommends allowing PerplexityBot and says that bot is not used to gather content for training. Anthropic's help page says blocking Claude-SearchBot may reduce your visibility in search results.
One rule covers all four: look for the search crawler's name, not the company's most famous bot.
Check 1: read your robots.txt
The file lives at the root of each address, and each subdomain has its own. If your shop sits on a different subdomain from your main site, check both. The file is a list of groups. Each group starts with one or more User-agent lines naming a crawler, followed by Allow and Disallow lines. Look for three things.
- A blanket block.
User-agent: *followed byDisallow: /shuts out every crawler that follows the rules. It is often left over from when the site was being built. - A named block. A group for OAI-SearchBot, PerplexityBot or Claude-SearchBot with
Disallow: /. - A named group that forgets something. Under the robots.txt standard, RFC 9309 of September 2022, a crawler that finds a group with its own name follows that group and ignores the
*group. A named group replaces the general rules. It does not add to them.
Here is our own file, fetched on 10 November 2026, shortened by two bots:
User-agent: *
Allow: /
Disallow: /blog/_drafts/
User-agent: GPTBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /Reading it: every crawler is allowed everywhere except one drafts folder. OAI-SearchBot isn't named, so it falls under the * group and is allowed. The file also shows the third point in action. GPTBot and PerplexityBot have their own group, so they skip the general group, including its one Disallow line. That is harmless here, because that folder is never published. On a site with a real private area it would be a hole.
Check 2: settings that write the rules for you
You may never have edited robots.txt and still have rules in it, because website builders and security services add them from a settings page.
- Squarespace. Under Settings, then Crawlers, there is a box labelled "Block known artificial intelligence crawlers". Squarespace's help page says it is unticked by default and lists the 26 bots it covers, including GPTBot, ClaudeBot and Google-Extended. On 10 November 2026 that list did not include OAI-SearchBot or PerplexityBot. So ticking the box does not take a site out of ChatGPT search.
- Cloudflare. Under Security, then Settings, filter by Bot traffic and open "AI Crawler and Agent access policies". There are three behaviours, Search, Agent and Training, each with its own allow or block setting. Many articles still say Cloudflare blocks AI crawlers by default, and the company did announce that direction on 1 July 2025. But Cloudflare's documentation, last updated on 2 November 2026, says new domains start with all three on Allow. A domain that has been on Cloudflare for longer may carry an earlier choice, so look instead of assuming. If Bot Preference Sync is switched on and any of the three is set to block, Cloudflare adds its own lines to the top of your robots.txt, which is how a file can contain rules nobody on your team wrote.
- Wix. The file is under SEO and GEO, then Tools and settings, then Robots.txt Editor. Wix's help article says the default file lets search engine bots reach all pages, and it mentions no separate setting for AI crawlers.
On any other builder or security plugin, search the settings for "crawlers", "bots" or "AI".
Check 3: is the door actually open?
A robots.txt file is a request, not a lock: the standard itself says the rules are not a form of access authorization. That cuts both ways. The file can say welcome while a firewall or bot protection refuses the visit.
If you're comfortable with a terminal, ask for your homepage while giving the crawler's name:
curl -I -A "OAI-SearchBot" https://example.com/A first line ending in 200 means the request got through. 403 means something refused it. We ran this against our own site on 10 November 2026 with the names OAI-SearchBot, GPTBot, PerplexityBot and Claude-SearchBot, and all four returned 200.
What this test can't see: it only catches rules that match on the crawler's name. Real crawlers arrive from the companies' own published addresses, and bot protection can treat them differently from your laptop. Perplexity's documentation warns that Cloudflare's Bot Fight Mode can still challenge its bots even after you add a rule to let them through.
If you're not technical, send one question to whoever hosts or maintains your site: "Do we block or challenge OAI-SearchBot, PerplexityBot, Claude-SearchBot or Googlebot anywhere, in robots.txt, the firewall, bot protection or a security plugin?" Ask them to search the server logs for OAI-SearchBot as well. If it visits and gets a 200, it's in.
Check 4: can it read what it finds?
A crawler that gets in still needs words. OpenAI's crawler page doesn't say whether its crawlers run JavaScript, so don't count on it. Open your homepage, right-click, choose View page source and search for the sentence that says what you do and where. If it's in the source, any crawler can read it. If your text only appears after scripts have run, raise it with whoever built the site.
This is also where being readable meets being worth quoting. When we tested how ChatGPT picks local businesses, every winner had written down, in plain words, the exact thing the customer asked for.
What to allow
For a business that wants customers to find it, the sensible default is to let the search crawlers in and to decide about training separately. That second choice doesn't affect whether you appear in answers. A file that stays open to search and opts out of OpenAI's training looks like this:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /If you add named groups for the search crawlers too, repeat inside each one any Disallow lines they should obey. And never rely on robots.txt to hide anything private. Put private pages behind a login.
After you change it
OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust for search results. Perplexity says up to 24 hours. Then be patient. Being readable is the entry ticket, not a recommendation. It gives the assistants the chance to find and quote you, and nothing more.
To see whether that chance turns into mentions, run the 20-minute test in how to check if ChatGPT recommends your business, and make sure the facts the crawlers find agree with your business listings. Our guide, Get Recommended by ChatGPT, local service businesses edition, takes it from there: a one-week audit across ChatGPT, Google's AI Mode and Perplexity, then a 30-day plan for your website, profiles and reviews.
Crawler names and defaults change, so re-read the vendors' own pages when you next touch this file. Neurova AI builds AI systems to healthcare standards from Eindhoven, and making a site readable for AI search is part of what we do for practices and service businesses. If you'd rather have it checked for you, tell us what you need.
Frequently asked questions
Which crawler does ChatGPT search use? OAI-SearchBot. OpenAI's crawler documentation says it is used to surface websites in ChatGPT's search features. GPTBot is a separate crawler that collects content that may be used to train OpenAI's models, and ChatGPT-User fetches a page when a person's question calls for it.
Does blocking GPTBot remove my website from ChatGPT? No. OpenAI says each setting is independent, so you can disallow GPTBot to opt out of training and still allow OAI-SearchBot to appear in ChatGPT search. The reverse also holds: allowing GPTBot does not put you into search answers if OAI-SearchBot is blocked.
How long does it take for a robots.txt change to reach ChatGPT? OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust for search results. Perplexity gives the same order of time, up to 24 hours. Being readable does not mean being recommended, so test again over the following weeks instead of the next morning.
Does robots.txt actually stop AI crawlers? It is a request, not a lock. The robots.txt standard, RFC 9309, says the rules are not a form of access authorization. Well-behaved crawlers follow it, but fetches started by a user, such as ChatGPT-User and Perplexity-User, may not. To enforce a block you need a firewall or a login.
Does Cloudflare block AI crawlers by default? Not on new domains, according to Cloudflare's documentation as updated on 2 November 2026: Search, Agents and Training all start on Allow. Cloudflare announced a move towards blocking by default on 1 July 2025, and a domain set up earlier may carry a different choice, so check the setting under Security, then Settings.