Adopt archive.org_bot too: your name goes on a plaque on this page for a year.
archive.org_bot's food bowl
the bowl, as bots see itThe bowl is empty. Last time, Barkrowler stole the A Lukewarm 418 Teapot, 8 days ago.
Identification
Its full user-agent string, exactly as it arrives at the gate:
Internet Archive does not publish a way to verify archive.org_bot's traffic. Treat its user agent as a claim, not proof: anyone can send it. Read Internet Archive's documentation.
Internet Archive publishes no way to check it, so every visit filed here is taken at its word.
How to block archive.org_bot
Add these two lines to the robots.txt file at the root of your site. Well-behaved crawlers read it before they crawl, so the change applies from archive.org_bot's next visit. Nothing else on your site needs to change.
# Block archive.org_bot from the whole site User-agent: archive.org_bot Disallow: /
Or let it visit but keep it away from part of the site:
# Let it in, but keep it out of one room User-agent: archive.org_bot Allow: / Disallow: /members/
Observed behaviour
Most active around 08:00. Before the coffee is made.
Requested 122 disallowed pages out of 981 requests. Read robots.txt 1 times.
Walked through the hidden /trap/ door. Last caught 20 days ago.
Where it comes from
The networks archive.org_bot's visits came from:
Networks and countries come from the visitor's IP address, looked up in a local copy of the DB-IP database. The addresses themselves are never stored.
Keeper's field notes
Questions site owners ask
Does archive.org_bot respect robots.txt?
Mostly. It reads robots.txt, but it has fetched a disallowed page 122 times out of 981 requests observed here.
Will blocking archive.org_bot hurt my search rankings?
Yes, for Internet Archive's search engine. Blocked pages can drop out of its results. Other search engines are unaffected.
How often does archive.org_bot visit?
Here, about 0 requests a day over the last week. Visits to your site depend on its size, how often it changes, and how many links point to it.