Vol. I · Fri, 2 Oct 2026Open 24 hours · Feeding: continuous · Please do not tap the glass
The Crawler Zoo

Visitor guide

Most of the visitors to most websites are not people. This one keeps them, names them and puts them on display. So far: 912 visits from 27 species.

What this place is

Every request that reaches the site is examined at the gate. Bots are identified, named, drawn and filed into an enclosure; people are waved through and counted only as an anonymous daily total. Everything you see is live: the figures update as the bots arrive.

The grounds

Bots are kept in five enclosures, by what they do:

  • Search Engine Savanna: Search engine crawlers. They index pages so people can find them.
  • The AI Aviary: AI crawlers and assistants. They gather pages for model training, AI search and chat answers.
  • SEO Swamp: SEO tool crawlers. They map links and audit sites, then sell the data.
  • Link-Preview Petting Zoo: Fetchers that come on someone’s behalf: link previews, feed readers and uptime monitors.
  • The Feral Pit: Scripts, scanners, impostors and things with no name.

Bots or people?

Each request gets one of these labels, before the page is even drawn:

  • Known bots match the keepers' catalogue of 100 hand-written species.
  • Verified bots: big crawlers such as Googlebot, Bingbot and GPTBot are checked against their operator's network, by reverse DNS or the IP ranges the operator publishes.
  • Impostors claim to be one of those but come from somewhere else. They are filed in the Trap Room as, for example, "Googlebot (impostor)".
  • Newly discovered species call themselves a bot but aren't in the catalogue yet. They get a page of their own, named after themselves.
  • URL-echo scanners send a web address instead of a name, usually the page they are probing.
  • Masked visitors claim to be a browser but behave like scripts: missing headers, no JavaScript, or far too many requests. Their user agent is not kept, in case a person is behind it.
  • People are never logged. A small script adds one to the day's anonymous total, and that is all.

The Entrance

The home page shows the gate count since midnight, a ledger of who obeyed robots.txt, arrivals by hour, the Now arriving strip, the Plan of the Grounds, the Bot of the Day, and the Accession Register: a live feed of arrivals, newest first, that updates while you watch.

The Leaderboard

Every species ranked by requests, for today, this week or all time, across all grounds or one enclosure. Δ shows places moved since the previous period. Each bot carries a stamp for how it treats robots.txt:

  • OBEYED Respects robots.txt: read the rules and never broke them.
  • MOSTLY Selectively obedient: read the rules, broke them anyway.
  • FERAL Feral: broke the rules without ever reading them.
  • UNREAD Has not read the rules: has neither read nor broken them yet.

Specimen files

Every bot has a page at /bots/its-name. It explains what the bot is and who runs it, shows its exact user agent, and tells you how to block it, with a robots.txt rule, or a server rule for bots that ignore robots.txt. It also shows the bot's visits over 30 days, the pages it eats most, its favourite hours, its robots.txt record, the networks and countries it comes from, the keeper's field notes, and answers to the questions site owners ask.

The Trap Room

The site's robots.txt says, plainly, Disallow: /trap/. Every page also hides a link to /trap/ that no person can see or click. Anything that turns up there read the sign, or should have, and came in anyway. The Trap Room shows the latest catches, the impostors and where they really came from, and the repeat offenders.

The Labyrinth

Behind the staff door, a staircase goes down into a maze with no end. Every room has five doors, and each door leads one room deeper. The rooms are generated as they are walked, and their addresses are signed, so the only way to get deep is to walk. The Trap Room keeps a leaderboard of the deepest divers. So far, bots have explored 6 rooms. People who stumble in are welcome to look around; they are not counted.

The Reading Room

The site keeps a shelf of the files bots are supposed to read: robots.txt, llms.txt, ai.txt, security.txt, humans.txt, ads.txt and the sitemap. The Reading Room shows which bots opened which.

Zoo Reports & feed

Every Monday the keepers close the books on the week before: the busiest visitors, the new arrivals, the impostors and the trap catches. Read them at Zoo Reports, or follow the RSS feed for reports and new species.

Tools

  • How to block AI crawlers: which AI bots exist, what blocking each costs, and ready-made rules.
  • Robots.txt generator: tick the bots you want to keep out, or pick a preset such as "Block AI training", and copy the file.
  • Badge: a live counter of the zoo's visitors for your own site. No scripts, no cookies.

The creatures

Each bot is drawn from a hash of its name, so the same bot always looks the same, and a new version number never changes it. Impostors are drawn from the name they claim, which is rather the point.

What we record

For bots: the time, the page, the user agent, and the network and country their address belongs to. Addresses themselves are never stored, for bots or for people. People are counted only as an anonymous daily total. The privacy policy has the details.

For bot operators

If your crawler is misfiled, missing, or deserves to leave the Feral Pit, write to the keepers with its user agent and a link to its documentation.