robots.txt (AI Context)

The robots.txt file has taken on new importance in the AI era as the primary mechanism for controlling access by AI crawlers. Traditional robots.txt usage focused on preventing search engine crawlers from accessing staging environments, duplicate content, or private sections. In the AI context, it is the critical control lever for granting or denying AI training and retrieval crawlers access to your content.

Major AI companies have registered specific user-agents for their crawlers: OpenAI uses GPTBot (for training) and OAI-SearchBot (for ChatGPT Search retrieval), Anthropic uses ClaudeBot (for Claude training), Google uses Google-Extended (for Gemini training), and Perplexity uses PerplexityBot (for retrieval). Each of these can be individually allowed or blocked via robots.txt directives, giving publishers fine-grained control over which systems can use their content.

Many site owners inadvertently block AI crawlers with overly broad robots.txt rules — for example, a Disallow: / for all user-agents blocks everyone, including AI retrieval bots that would otherwise source traffic back. Other sites explicitly block AI training crawlers but allow retrieval crawlers, making a deliberate distinction between training data use (no attribution) and real-time citation (with traffic attribution).

For AI SEO strategy, the standard recommendation is to allow retrieval crawlers (OAI-SearchBot, PerplexityBot) while making an informed decision about training crawlers (GPTBot, ClaudeBot, Google-Extended). Sites that want to maximize AI visibility should ensure no relevant AI crawlers are blocked, and should regularly audit robots.txt as new crawlers are registered by emerging AI platforms.

An online publisher audits its robots.txt, discovers a legacy Disallow: / rule was blocking all unrecognized crawlers, removes the overly broad rule, adds specific Allow directives for OAI-SearchBot and PerplexityBot, and begins appearing in ChatGPT Search responses within four weeks.