Tools

Robots.txt
generator

Choose who may come in. Tick the bots you want to keep out, or start from a preset, and copy the file to the root of your site. Not sure which to pick? Read how to block AI crawlers.

PresetsBlock AI training Block all AI Block SEO tools Search engines only Clear

Search Engine SavannaGooglebot Google
Google's main crawler.GoogleOther Google
A general-purpose Google crawler used by product teams for research and one-off fetches.Mediapartners-Google Google
Google AdSense's crawler.AdsBot-Google Google
Checks the quality of landing pages used in Google Ads.Bingbot Microsoft
Microsoft's crawler for Bing, whose index also powers several other search engines and Copilot answers.Applebot Apple
Apple's crawler for Siri and Spotlight suggestions.DuckDuckBot DuckDuckGo
DuckDuckGo's own crawler, used alongside results from partners.YandexBot Yandex
Yandex's search crawler.Baiduspider Baidu
Baidu's search crawler.PetalBot Huawei
Huawei's crawler for Petal Search.SeznamBot Seznam.cz
The crawler for Seznam, the Czech search engine.archive.org_bot Internet Archive
The Internet Archive's crawler.Google-InspectionTool Google
Google's testing crawler, used by Search Console's URL Inspection and the Rich Results Test.Storebot-Google Google
Google's shopping crawler.Google-Display-Ads-Bot Google
A user agent that names Google's display advertising.YisouSpider Shenma (Alibaba)
The crawler for Shenma, Alibaba's mobile search engine in China.Sogou web spider Sogou (Tencent)
The crawler for Sogou, a Chinese search engine now owned by Tencent.coccocbot Cốc Cốc
The crawler for Cốc Cốc, a Vietnamese search engine and browser.MojeekBot Mojeek
The crawler for Mojeek, an independent British search engine with its own index.Qwantbot Qwant
The crawler for Qwant, a French search engine that promises not to track its users.SeekportBot Seekport
The crawler for Seekport, a German search engine.

The AI AviaryGPTBot OpenAI
OpenAI's web crawler.OAI-SearchBot OpenAI
OpenAI's search crawler.ChatGPT-User OpenAI
Fetches a page when a ChatGPT user asks about it.ClaudeBot Anthropic
Anthropic's web crawler.Claude-SearchBot Anthropic
Anthropic's search crawler.Claude-User Anthropic
Fetches a page when a Claude user asks about it.CCBot Common Crawl
Common Crawl's crawler.PerplexityBot Perplexity
Perplexity's crawler, which indexes pages for its AI answer engine.Perplexity-User Perplexity
Fetches pages when a Perplexity user asks a question that needs them.Bytespider ByteDance
ByteDance's crawler, used to gather data for its AI products.Meta-ExternalAgent Meta
Meta's crawler for AI training and product indexing.Amazonbot Amazon
Amazon's crawler, used to improve Alexa and other Amazon services.Diffbot Diffbot
Extracts structured data from pages for the Diffbot knowledge graph, which AI companies license.AgentTrustBot AgentTrust
A crawler from AgentTrust, which gives its purpose as "AI Agent Discovery": looking for sites and services that AI agents can use.cohere-ai Cohere
Cohere's crawler, which collects public pages that may be used to train Cohere's language models.AI2Bot Allen Institute for AI
The crawler of the Allen Institute for AI (Ai2), a non-profit that builds open AI models and datasets.YouBot You.com
The crawler for You.com, an AI search and chat assistant.DuckAssistBot DuckDuckGo
DuckDuckGo's crawler for DuckAssist, its AI-written answers.MistralAI-User Mistral AI
Fetches a page when someone using Mistral's Le Chat assistant asks about it.ImagesiftBot Hive
An image crawler run by Hive, an AI company, for its ImageSift image search.Google-Extended Google
Controls whether Google may use your pages to train Gemini. It is not a crawler: Googlebot reads this rule.Applebot-Extended Apple
Controls whether Apple may use your pages to train its AI models. Applebot reads this rule.

SEO SwampAhrefsBot Ahrefs
Ahrefs' crawler, which maps links between websites for its SEO tools.SemrushBot Semrush
Semrush's crawler for backlink and site-audit data.MJ12bot Majestic
Majestic's distributed crawler, run on volunteers' computers, for its link index.DotBot Moz
Moz's crawler for its link index.DataForSeoBot DataForSEO
DataForSEO's crawler, which collects SEO data sold to other tools.BLEXBot WebMeUp
WebMeUp's backlink crawler.Pandalytics DomainsBot
The crawler behind DomainsBot, which collects data about websites and domain names for its domain-search and analytics products.DomainArrivalsBot DomainArrivals
A crawler from DomainArrivals, a service that follows newly registered domains as they appear on the web.SerpstatBot Serpstat
Serpstat's crawler, which tracks backlinks for its SEO platform.Barkrowler Babbar
The crawler for Babbar, a French SEO tool that maps links and the topics of pages.Screaming Frog SEO Spider Screaming Frog
A desktop SEO crawler.AwarioBot Awario
The crawler for Awario, a social-listening tool that tracks where brands and keywords are mentioned online.MegaIndex MegaIndex
The crawler for MegaIndex, a Russian SEO and backlink-analysis service.

Link-Preview Petting Zoofacebookexternalhit Meta
Fetches the title and image of a page when someone shares a link on Facebook, Instagram or Messenger.TelegramBot Telegram
Fetches link previews in Telegram chats.Twitterbot X
Fetches link previews when a URL is posted on X.Slackbot Slack
Unfurls links pasted into Slack.Discordbot Discord
Generates link embeds in Discord.LinkedInBot LinkedIn
Generates link previews on LinkedIn.WhatsApp Meta
Fetches link previews inside WhatsApp chats.Pinterestbot Pinterest
Fetches pages and images when someone saves a link to Pinterest.redditbot Reddit
Fetches link previews for posts on Reddit.Mastodon Fediverse servers
Fetches link previews for Mastodon servers.Bluesky Bluesky
Fetches the title and image for link cards when someone posts a link on Bluesky.Iframely Iframely
A link-preview service used by many apps and publishing platforms to turn a pasted link into a rich embed.Embedly Embedly
A link-preview and embed service used by apps and websites to show previews of shared links.SkypeUriPreview Microsoft
Fetches link previews when someone shares a link in Microsoft Teams or Skype.Feedly Feedly
Fetches RSS feeds for Feedly, a news reader.Inoreader Inoreader
Fetches RSS feeds for Inoreader, a news reader.NewsBlur NewsBlur
Fetches RSS feeds for NewsBlur, an independent news reader.UptimeRobot UptimeRobot
An uptime monitor.Pingdom SolarWinds
An uptime and speed monitor from SolarWinds.

The Feral PitWget Unknown
GNU Wget, a command-line tool for downloading files and whole websites.Scrapy Unknown
Scrapy, a Python framework for building web scrapers.FossickBot fossick.bot
A small crawler that identifies itself only as FossickBot, with a link to fossick.bot.ChamberBot Unknown
A crawler with no domain of its own: its information page lives on a nip.io address, which maps straight to a server’s IP address.srchs-research-bot An individual researcher
A small research crawler that describes its work as "public small-business domain research".

robots.txtDownload

# robots.txt made with The Crawler Zoo
# https://crawlerzoo.com/robots-txt-generator
#
# Nothing blocked: every crawler may read the whole site.

User-agent: *
Allow: /

Googlebot: Blocking Googlebot removes your pages from Google Search. To opt out of Gemini training only, block the Google-Extended token instead.

Mediapartners-Google: If your site shows AdSense ads, do not block it. Without it, AdSense cannot read your pages and ads become less relevant.

AdsBot-Google: AdsBot ignores the "User-agent: *" group. You must name it to block it.

Bingbot: Blocking Bingbot also removes you from search engines that use the Bing index, such as DuckDuckGo and Yahoo.

Applebot: To stay in Siri and Spotlight but opt out of Apple AI training, block the Applebot-Extended token instead.

GPTBot: Blocking GPTBot does not remove you from ChatGPT search. That is OAI-SearchBot, which has its own token.

ChatGPT-User: Because it acts for a user, OpenAI says robots.txt rules may not apply to it.

ClaudeBot: Anthropic also runs Claude-SearchBot (search) and Claude-User (user requests). Each has its own token.

Perplexity-User: Because it acts for a user, Perplexity says it generally ignores robots.txt.

Bytespider: Many site owners report it does not always follow robots.txt. Check its Trap Room record, and consider a server-level block too.

facebookexternalhit: Blocking it breaks link previews on Meta apps. Meta says it may skip robots.txt for security and integrity checks.

Google-InspectionTool: Blocking it stops Search Console from testing your pages. It does not affect Google Search itself.

Storebot-Google: If you sell things, blocking it can keep your products out of Google Shopping.

cohere-ai: Cohere uses two names, cohere-ai and cohere-training-data-crawler. The rule blocks both.

Screaming Frog SEO Spider: It reads robots.txt by default, but its user can switch that off.

UptimeRobot: It only visits pages someone has asked it to watch. To stop it, the person who set up the monitor has to remove it.

Pingdom: It only visits pages someone has asked it to watch.

Wget: Wget reads robots.txt when it downloads a site recursively, but not for single files.

Scrapy: New Scrapy projects read robots.txt by default, but their authors can turn that off.

Google-Extended: Blocking Google-Extended does not affect Google Search.

Applebot-Extended: Blocking Applebot-Extended does not remove you from Siri or Spotlight.

Not every bot reads robots.txt

Scripts, scanners and impostors ignore it. To keep them out, block them at your web server or firewall. Their specimen files have ready-made rules: HeadlessChrome, python-requests, curl, Go-http-client, zgrab, CensysInspect, Expanse, InternetMeasurement, Nmap Scripting Engine, Nuclei, masscan, okhttp, axios, python-httpx, aiohttp, Python-urllib, Java, libwww-perl, URL-echo scanner, (no user agent).

Robots.txtgenerator

Robots.txt
generator