User-agent: * Disallow: /trap/ Disallow: /api/ # Google ignores "*" for its ads crawlers, so name them. User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile Disallow: /trap/ Disallow: /api/ Sitemap: https://crawlerzoo.com/sitemap.xml