Vol. I · Fri, 2 Oct 2026Open 24 hours · Feeding: continuous · Please do not tap the glass
Plate LXVII0x26534104UNREAD
Kingdom Automata›Phylum HTTP›Order Crawlers›Family SEO Crawlers›Species Screaming Frog SEO Spider
Specimen file · SEO Swamp

Screaming Frog SEO Spider

Rana clamans

What is Screaming Frog SEO Spider?

A desktop SEO crawler. Someone runs it from their own computer to audit a site, often their own, sometimes a competitor's. It is operated by Screaming Frog and identifies itself as Screaming Frog SEO Spider.

Temperament: loud. The only resident of the Swamp that is technically a frog.

Operator
Screaming Frog
Conservation status
Has not read the rules
First seen
Not yet
Last seen
Not yet
Visits today
0
This week
0
All time
0
Caught in the trap
Never
§ I.

Identification

Its full user-agent string, exactly as it arrives at the gate:

User-Agent
$ Screaming Frog SEO Spider/20.0

Screaming Frog does not publish a way to verify Screaming Frog SEO Spider's traffic. Treat its user agent as a claim, not proof: anyone can send it.

§ II.

How to block Screaming Frog SEO Spider

Add these two lines to the robots.txt file at the root of your site. Well-behaved crawlers read it before they crawl, so the change applies from Screaming Frog SEO Spider's next visit. Nothing else on your site needs to change.

robots.txt
# Block Screaming Frog SEO Spider from the whole site
User-agent: Screaming Frog SEO Spider
Disallow: /

It reads robots.txt by default, but its user can switch that off.

Or let it visit but keep it away from part of the site:

# Let it in, but keep it out of one room
User-agent: Screaming Frog SEO Spider
Allow: /
Disallow: /members/
§ III.

Observed behaviour

Visits, last 30 dayspeak 1 / day
3 Sept 2026today
Diet · pages most often eaten

Nothing on record in the last 30 days.

Visiting hours
00h – 11h12h – 23h
robots.txt compliance
100.0%

Requested 0 disallowed pages out of 0 requests. Never read robots.txt.

Trap Room record
Clean

Has never followed the hidden link to /trap/. Either well trained or very lucky.

§ IV.

Where it comes from

The networks Screaming Frog SEO Spider's visits came from:

Nothing on record in the last 30 days.

Networks and countries come from the visitor's IP address, looked up in a local copy of the DB-IP database. The addresses themselves are never stored.

§ V.

Questions site owners ask

Does Screaming Frog SEO Spider respect robots.txt?

We can't say yet. It has not fetched robots.txt here, and it has not touched a disallowed page either.

Will blocking Screaming Frog SEO Spider hurt my search rankings?

No. SEO crawlers build tools for marketers. Blocking them hides your links from those tools, not from search engines.

How often does Screaming Frog SEO Spider visit?

Here, about 0 requests a day over the last week. Visits to your site depend on its size, how often it changes, and how many links point to it.

Also in SEO Swamp