# robots.txt for AI Crawlers: GPTBot, ClaudeBot, Perplexity (2026) > Your robots.txt file is how you allow or block AI crawlers like GPTBot, ClaudeBot, and PerplexityBot. Learn what to put in it, how it differs from search bots, and how Sabilytics checks it. - Canonical: https://www.sabilytics.com/blog/robots-txt-for-ai-crawlers - Markdown: https://www.sabilytics.com/blog/robots-txt-for-ai-crawlers.md - Author: Ayodele S. Adebayo (Founder, Sabilytics) - Section: AI Discoverability - Published: 2026-08-14 **robots.txt** is a public text file at `https://yourdomain.com/robots.txt`. Crawlers read it first to learn what they may fetch. AI assistants have their own bot names. If those names are blocked, or if you `Disallow: /` for everyone, models that obey the file will not learn your pages from your site. This guide covers how to allow (or intentionally refuse) AI crawlers, how that interacts with Googlebot, and how Sabilytics scores the **robots.txt / AI crawlers** check. If you are new to the broader idea, start with [What is AI Discoverability?](/blog/what-is-ai-discoverability). Search engines use the same file for a slightly different check: [Search Discoverability](/blog/what-is-search-discoverability). ## TL;DR - One `robots.txt` can allow Google and still refuse (or allow) AI, because vendors use named bots. - Training, search answers, and live browsing are different user-agents. Blocking GPTBot does not block [OAI-SearchBot](https://developers.openai.com/api/docs/bots). - If you want to be cited in assistant search, do not accidentally disallow OAI-SearchBot, Claude-SearchBot, or PerplexityBot. - A missing file, or `User-agent: *` plus `Disallow: /` on production, is the usual self-own. ## Why AI crawlers need their own lines A typical file talks to `User-agent: *` (everyone) and maybe Googlebot. Assistant vendors also send named crawlers, and they split jobs: - **OpenAI:** [GPTBot](https://developers.openai.com/api/docs/bots) (training), **OAI-SearchBot** (ChatGPT search answers), **ChatGPT-User** (a user or agent fetching a URL) - **Anthropic:** [ClaudeBot](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) (training), **Claude-SearchBot** (search), **Claude-User** (live browse) - **PerplexityBot** - **Google-Extended** (Gemini / Google's generative use, separate from Googlebot) If `*` allows `/`, many bots follow that. If `*` disallows `/`, you have locked the door for everyone, including assistants, unless you add a more specific allow for a named agent (crawler behavior still varies). If you copied a snippet that lists `GPTBot` with `Disallow: /` "to be safe," you opted out of that vendor's training or browsing crawl on purpose. There is no universal law that every model honors robots.txt the same way. Treating the file as your public policy is still the right default: say what you want, keep it consistent with [`llms.txt`](/blog/what-is-llms-txt), and do not leave a blanket block from staging on production. ## Allow vs block: make it a product decision **Allow** if you want assistants to fetch docs, pricing, and the homepage when they try. That is the usual choice for a public product site. **Block named AI crawlers** if you do not want those vendors to use your content in the ways their docs describe. That is valid. Do it explicitly. Do not expect to block GPTBot and still be represented in ChatGPT search: that path uses OAI-SearchBot. **Block everything** (`User-agent: *` + `Disallow: /`) only for private apps, staging, and sites that must not be crawled. A marketing homepage with that rule is a self-own. Sabilytics does not tell you that allowing AI is morally required. It tells you whether your current file *looks* like it welcomes major AI crawlers, blocks them, or never mentions them. ## A production file that stays readable Keep one file. Search bots and AI bots can share the same disallow list for private routes (`/dashboard/`, `/api/`, `/invite/`). ```text User-agent: * Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: Googlebot Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: GPTBot Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: ChatGPT-User Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: OAI-SearchBot Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: ClaudeBot Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: Claude-SearchBot Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: Claude-User Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: PerplexityBot Allow: / Disallow: /dashboard/ Disallow: /api/ User-agent: Google-Extended Allow: / Disallow: /dashboard/ Disallow: /api/ Sitemap: https://example.com/sitemap.xml ``` Replace the domain and the private paths with yours. Mentioning named AI agents is how you make the policy obvious to humans and to checkers. A lone `User-agent: *` / `Allow: /` can be enough in practice, but it is easy to misread, and Sabilytics will warn if AI crawlers are never named. To **refuse** a vendor, keep the rest, and for that agent only: ```text User-agent: GPTBot Disallow: / ``` Be consistent. Do not publish a glowing `llms.txt` while blocking every bot that might fetch it. ## What Sabilytics looks for The AI Discoverability finding id is `robots_ai_crawlers`. During a scan, Sabilytics fetches `/robots.txt` and parses groups of `User-agent` plus `Disallow` rules. It treats these names as major AI crawlers (matching is case-insensitive, substring-style): GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, Anthropic-AI, PerplexityBot, Google-Extended, Bytespider, CCBot. Outcomes you will see: - **No robots.txt found** (warning). Engines and assistants lack a policy and often miss your sitemap pointer. - **AI crawlers appear blocked**, or `User-agent: *` disallows `/` (fail). Assistants that honor the file will not read public pages. - **robots.txt does not mention AI crawlers** (warning). Crawl might still be allowed via `*`, but the policy is unclear. - **robots.txt allows AI crawlers** (pass). Named agents are present and not fully disallowed. Search Discoverability uses the **same file** for a general crawl check (`robots_crawl`): missing file, or a blanket block of `*`, versus allowing search engines. Fix production once. Both scores should move. ## Common mistakes **Staging policy on production.** Preview deploys often disallow all. The same `robots.ts` then ships to the apex. **Blocking GPTBot but allowing `*`.** Training and search are different names (`ChatGPT-User`, `OAI-SearchBot`, `Claude-SearchBot`). Read the vendor's current bot list when you care. **No Sitemap line.** robots.txt is also how you advertise `/sitemap.xml`. Both AI and search checks care. **HTML error page at /robots.txt.** A 200 with a Next.js 404 document is not a robots file. Confirm `Content-Type` is text and the body starts with `User-agent`. **llms.txt without crawl access.** A briefing nobody may fetch is a note in a locked drawer. ## How to publish it On Next.js, `app/robots.ts` (or `app/robots.txt`) can generate the file. Static hosts can put `public/robots.txt` at the site root. 1. Deploy. 2. Open `https://yourdomain.com/robots.txt` in a private window. 3. Confirm private routes are disallowed and the homepage is not. 4. Re-run [AI Discoverability](/ai-discoverability) and [Search Discoverability](/search-discoverability). Update the file when you add app routes you must keep private, or when you change your mind about a vendor. ## Common myths **"Blocking AI bots keeps me out of answers but in search."** Googlebot and Google-Extended are different. You can allow search and refuse generative use, or the reverse, if you set agents separately. Test the file you actually shipped. **"User-agent: * is enough, naming bots is noise."** Often `*` is enough for crawlers that honor it. Naming is documentation, and it is what the Sabilytics AI check looks for so you do not silently inherit a block list. **"robots.txt hides private pages from humans."** It does not. It is a crawl hint. Authentication still belongs on dashboards and APIs. **"I need a different robots.txt for ChatGPT than for Google."** One file, multiple `User-agent` groups. That is the format. **"A pass means ChatGPT will cite me."** A pass means crawlers are not obviously forbidden. Citation is a separate, vendor-side choice. See the myths in the [AI Discoverability guide](/blog/what-is-ai-discoverability). ## Frequently asked questions ### Which bots does Sabilytics care about? The parser looks for common AI user-agents listed above. Vendor names change. If you allow the ones you care about and you do not blanket-disallow `*`, you are in the intended shape. ### Should I allow Bytespider and CCBot? That is a policy choice (ByteDance / common crawl style bots). Sabilytics includes them in "mentions AI crawlers" matching. Allowing GPTBot and ClaudeBot while omitting others can still pass if you do not block majors. Read the finding copy on your scan. ### Do I need to allow AI to rank on Google? No. Search crawl is Googlebot (and friends). AI Discoverability is a sibling score. You can pass search and fail AI, or the other way around. ### Can I generate robots.txt with an agent? Yes. Give it your real private routes. Do not invent `Disallow` rules for URLs you want public. Sabilytics fix prompts are written for that workflow. ### Where should I start if the check fails? If the file is missing, publish one with `Allow: /`, private `Disallow`s, named AI agents you want, and a Sitemap. If it blocks `*`, remove that for production. Re-scan. ## Check your robots.txt on a real domain Policy only counts if the public URL returns it. Sabilytics AI Discoverability includes the robots / AI crawlers finding, next to sitemap, `llms.txt`, and homepage identity. You get impact text and a fix hint you can paste into a coding assistant. When you are ready, [run a free AI Discoverability check](/ai-discoverability) and confirm `/robots.txt` matches the site you meant to ship. ## Sources - [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots) - [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)