How to block
AI crawlers

"AI crawler" covers three different jobs. They have different names, so you can choose which to keep out.

Copy one of these into the robots.txt file at the root of your site, or open it in the generator to adjust it.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: AI2Bot
User-agent: Diffbot
Disallow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: AI2Bot
User-agent: Diffbot
User-agent: ImagesiftBot
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: YouBot
User-agent: DuckAssistBot
User-agent: Amazonbot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /members/
Disallow: /drafts/

These two are not crawlers. They are names you can use in robots.txt to tell Google and Apple not to use your pages for their AI models. Googlebot and Applebot read the rule and carry on crawling for search, so blocking them keeps you in Google Search, Siri and Spotlight while opting out of Gemini and Apple Intelligence training.

Because they never visit, a server rule cannot block them. Only robots.txt can.

robots.txt is a request, not a lock. It works because the big companies choose to respect it. Three things it cannot stop:

• Bots that ignore it. In the last 30 days, 28 visitors walked through a door this site's robots.txt marks as off-limits. The Trap Room lists them.
• Impostors. Anyone can call their scraper GPTBot. Real crawlers come from their operators' networks, and several publish their IP ranges. This zoo checks every visit; failures are filed as impostors, like GPTBot (impostor).
• Data already collected. Blocking works from the next visit. It does not remove pages that were fetched before.

For bots that ignore robots.txt, refuse them at your web server. A rule like this in nginx returns "403 Forbidden" to any listed name:

map $http_user_agent $ai_bot {
    default 0;
    ~*(GPTBot|ClaudeBot|CCBot|Bytespider|meta-externalagent|cohere|AI2Bot|Diffbot) 1;
}

server {
    # ...
    if ($ai_bot) { return 403; }
}

User agents are easy to fake, so a scraper can simply change its name. Pair this with rate limiting. If your site sits behind Cloudflare, its dashboard has a one-click switch to block known AI crawlers. (The zoo leaves it off, for obvious reasons.)

• llms.txt is a plain-text map of your site written for AI models. It invites them in rather than keeping them out. The zoo's llms.txt.
• ai.txt (from Spawning) states what AI may use your site for. Few crawlers read it yet.
• TDMRep is a W3C community standard for reserving text-and-data-mining rights, which matters under EU copyright rules. It uses a /.well-known/tdmrep.json file or a tdm-reservation header.
• "noai" meta tags are not a standard. A few services honour them; most crawlers do not.

The Reading Room shows which bots actually open these files.

Open https://your-site/robots.txt in a browser to confirm the file is live. Then watch your server logs for the names you blocked: well-behaved bots will fetch robots.txt and stop requesting your pages, usually within a day. Any that keep coming are ignoring the rule, and belong in a server block.