AdsBot-Google: AdsBot ignores the "User-agent: *" group. You must name it to block it.
Applebot: To stay in Siri and Spotlight but opt out of Apple AI training, block the Applebot-Extended token instead.
Applebot-Extended: Blocking Applebot-Extended does not remove you from Siri or Spotlight.
ArnorBot: ArnorBot obeys robots.txt and slows down when a site returns 429 or 503.
Bingbot: Blocking Bingbot also removes you from search engines that use the Bing index, such as DuckDuckGo and Yahoo.
BitsExplorerBot: It obeys robots.txt under the name bitsexplorer, and stays away if it can’t read robots.txt. BitsExplorer also has a removal form at bitsexplorer.com/removal.
Bravebot: Brave’s crawler page says it follows the robots.txt rules you set for Googlebot, and it doesn’t name a separate Bravebot token, so this rule may not work. To keep a page out of Brave Search, use a noindex tag.
BufferLinkPreviewBot: It honours robots.txt. Blocking it means Buffer posts that link to your site will have no preview.
Bytespider: Many site owners report it does not always follow robots.txt. Check its Trap Room record, and consider a server-level block too.
ChatGPT-User: Because it acts for a user, OpenAI says robots.txt rules may not apply to it.
ClaudeBot: Anthropic also runs Claude-SearchBot (search) and Claude-User (user requests). Each has its own token.
ClueWeb-Crawler: CMU says the crawler honours all robots.txt rules, including Crawl-delay. It doesn’t name a token; ClueWeb-Crawler is the name in its user agent.
cohere-ai: Cohere uses two names, cohere-ai and cohere-training-data-crawler. The rule blocks both.
ExaSearchBot: Exa's search is also used by other apps through its API, so blocking ExaSearchBot keeps your pages out of those results too.
facebookexternalhit: Blocking it breaks link previews on Meta apps. Meta says it may skip robots.txt for security and integrity checks.
fiperbot: Fiper doesn’t say whether it reads robots.txt. It asks sites that want it to stop to return a 404 to requests with “fiperbot” in the user agent.
FossickBot: FossickBot reads robots.txt first and obeys Crawl-delay. Its operator also takes opt-out requests at abuse@fossick.bot.
Google-Extended: Blocking Google-Extended does not affect Google Search.
Google-InspectionTool: Blocking it stops Search Console from testing your pages. It does not affect Google Search itself.
Googlebot: Blocking Googlebot removes your pages from Google Search. To opt out of Gemini training only, block the Google-Extended token instead.
GPTBot: Blocking GPTBot does not remove you from ChatGPT search. That is OAI-SearchBot, which has its own token.
KeenableBot: KeenableBot follows robots.txt (RFC 9309). It also obeys noindex and nofollow in meta robots tags and the X-Robots-Tag header.
Mediapartners-Google: If your site shows AdSense ads, do not block it. Without it, AdSense cannot read your pages and ads become less relevant.
Mediatoolkitbot: It respects robots.txt, including Crawl-delay. Complaints go to complaints@determ.com.
Meta-WebIndexer: Meta lists Meta-WebIndexer among the crawlers you control with robots.txt. Blocking it may keep your pages out of Meta AI’s answers. It doesn’t affect Meta-ExternalAgent, which Meta uses for AI training.
MetaRabbitBot: It obeys robots.txt and won’t fetch a page if it can’t read robots.txt. Email support@zumistudios.com to have a domain blocked and its cached data removed.
NetAPI: NetAPI follows robots.txt (RFC 9309).
Perplexity-User: Because it acts for a user, Perplexity says it generally ignores robots.txt.
Pingdom: It only visits pages someone has asked it to watch.
QlyzeBot: It obeys robots.txt and Crawl-delay. Email bot@qlyze.io to exclude a domain for good.
Scrapy crawler: New Scrapy projects read robots.txt by default, but their authors can turn that off.
Screaming Frog SEO Spider: It reads robots.txt by default, but its user can switch that off.
SEOJuice-SearchBot: It obeys robots.txt, including Crawl-delay, and treats anything closed to Googlebot as closed to itself too. Email hello@seojuice.com for an immediate stop.
SofyaBot: SofyaBot obeys robots.txt, including Crawl-delay.
SpiderLing: It respects robots.txt and waits at least 20 seconds between requests to the same site.
Storebot-Google: If you sell things, blocking it can keep your products out of Google Shopping.
trendictionbot: trendictionbot obeys robots.txt, but a change can take up to five days to apply.
UnGovrBot: UnGovr says its site-scan tool reads robots.txt as UnGovrBot before every request. It doesn’t say the same for its general crawler. Contact: crawler@ungovr.org.
UptimeRobot: It only visits pages someone has asked it to watch. To stop it, the person who set up the monitor has to remove it.
Walsh-Research: It obeys robots.txt (RFC 9309), including Crawl-delay, but treats a feed you advertise as an invitation: a wildcard Disallow won’t stop it polling your feed. Name it in robots.txt, or email j@wal.sh to opt out with no questions asked.
Wget command: Wget reads robots.txt when it downloads a site recursively, but not for single files.