Excessive Crawling from "FriendlyCrawler"


I’m new here (first post) and I just wanted to post a heads-up — I’ve seen a LOT of requests (every 10 seconds) from user agent:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15

…and it doesn’t seem to be friendly at all. I’ve been blocking it this morning and my WAF log now shows >2500 entries, in about 12 hours. It’s coming from AWS and I filed an abuse report with Amazon because it made my small shared hosting account’s “resource warning” light up.

If there’s a better category for this I hope a moderator can move it.


1 Like

We have seen the same - but we block all AWS anyway

Anyone can send anything as a User Agent - and there are a lot of so-called research and friendly User Agents that are just trawling for vulnerabilities

1 Like

The reply from AWS was rather sobering: requesting a lot of information that I don’t even know how to provide — or not in a reasonable amount of time. My time of course. I’m just a guy with a website…

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.