Vol. I · Sun, 15 Nov 2026Open 24 hours · Feeding: continuous · Please do not tap the glassPatrons☕ Keepers' coffee
Plate CX0x357B42BBOBEYED
Kingdom Automata›Phylum HTTP›Order Crawlers›Family Search Engines›Species ClueWeb-Crawler
Specimen file · Search Engine Savanna

ClueWeb-Crawler

Discipulus mellonensis

What is ClueWeb-Crawler?

An academic crawler from Carnegie Mellon University’s Language Technologies Institute. It collects public web pages for ClueWeb, a large web collection used in research. It reads public HTML only and says it avoids personal data. It is operated by Carnegie Mellon University and identifies itself as ClueWeb-Crawler.

Temperament: studious. Reading the web for science. Will cite this zoo, probably.

Operator
Carnegie Mellon University
Conservation status
Respects robots.txt
First seen
21 Oct 2026
Last seen
1 h ago
Visits today
2
This week
50
All time
128
Caught in the trap
Never

Nobody has adopted ClueWeb-Crawler yet. Adopt ClueWeb-Crawler: your name goes on a plaque on this page for a year.

ClueWeb-Crawler's food bowl

the bowl, as bots see it

The bowl is empty. Last time, Masked visitor stole the A Lukewarm 418 Teapot, 7 days ago.

5 of 5 tokens left today
§ I.

Identification

Its full user-agent string, exactly as it arrives at the gate:

User-Agent
$ ClueWeb-Crawler/1.0 (+https://boston.lti.cs.cmu.edu/CMU-ClueWeb-Crawler/; mailto:cmu-clueweb-crawler@andrew.cmu.edu)

Carnegie Mellon University does not publish a way to verify ClueWeb-Crawler's traffic. Treat its user agent as a claim, not proof: anyone can send it. Read Carnegie Mellon University's documentation.

Identity check · logged visits, last 30 days
IdentityCannot be verified

Carnegie Mellon University publishes no way to check it, so every visit filed here is taken at its word.

§ II.

How to block ClueWeb-Crawler

Add these two lines to the robots.txt file at the root of your site. Well-behaved crawlers read it before they crawl, so the change applies from ClueWeb-Crawler's next visit. Nothing else on your site needs to change.

robots.txt
# Block ClueWeb-Crawler from the whole site
User-agent: ClueWeb-Crawler
Disallow: /

CMU says the crawler honours all robots.txt rules, including Crawl-delay. It doesn’t name a token; ClueWeb-Crawler is the name in its user agent.

Or let it visit but keep it away from part of the site:

# Let it in, but keep it out of one room
User-agent: ClueWeb-Crawler
Allow: /
Disallow: /members/
§ III.

Observed behaviour

Visits, last 30 dayspeak 17 / day, 10 Nov 2026
17 Oct 2026today
Diet · pages most often eaten
/robots.txt26
/leaderboard12
/6
/coffee6
/adopt6
Visiting hours
00h – 11h12h – 23h

Most active around 05:00. The keepers are asleep. It knows.

robots.txt compliance
100.0%

Requested 0 disallowed pages out of 128 requests. Read robots.txt 26 times.

Trap Room record
Clean

Has never followed the hidden link to /trap/. Either well trained or very lucky.

§ IV.

Where it comes from

The networks ClueWeb-Crawler's visits came from:

Networks · 1 on file, last 30 days
Carnegie Mellon University AS9100%
Countries
United States US100%

Networks and countries come from the visitor's IP address, looked up in a local copy of the DB-IP database. The addresses themselves are never stored.

§ V.

Keeper's field notes

15 Nov 202611:17. Arrived without announcement. Consumed / and /robots.txt. Departed.
15 Nov 2026Observed reading robots.txt in full. Complied with every word.
10 Nov 2026Visited 17 times in a single day. Keeper unable to establish why.
21 Oct 2026First recorded at the gate. Entered in the register as Discipulus mellonensis.
§ VI.

Questions site owners ask

Does ClueWeb-Crawler respect robots.txt?

Yes. In 128 requests observed here it has read robots.txt and never fetched a disallowed page.

Will blocking ClueWeb-Crawler hurt my search rankings?

Yes, for Carnegie Mellon University's search engine. Blocked pages can drop out of its results. Other search engines are unaffected.

How often does ClueWeb-Crawler visit?

Here, about 7 requests a day over the last week. Visits to your site depend on its size, how often it changes, and how many links point to it.

Also in Search Engine Savanna