This appears to be some type of scraper which drops by several times an hour and parses every page on my site. By rotating through (what appears to be) a massive pool of good reputation IP’s from Verizon, Comcast, AT&T, etc, AND. . . By switching to a ‘’different’ IP for each page access, it can easily circumvent fail2ban, rate limiting and the Apache RBL’s.
So I figured I’d give Cloudflare a try with a free account. Evidently, this insidious bot appears to have no problem bypassing the CF (I’m under attack mode). The first thing that threw me off was that this bot continued to hit my pages, yet. . . The originating IP addresses were not appearing in Cloudflares firewall activity log. Hmmm
At second glance and by observing timestamps BETWEEN my server logs, AND CF’S firewall activity log, I could see that, in fact IP’s WERE hitting Cloudflare, but. . . They were NOT matching the IP’s in my server logs. Let’s look at a sample:
CF Firewall activity log:
Sep 13, 2023 2:24:04 AM
But 220.127.116.11 is NOT what hit my site. Observe what DID hit my site, and what is appearing in my server log:
A visitor from pool-74-99-177-198.hrbgpa.fios.verizon.net (18.104.22.168)
arrived without a referring URL,
and visited http://mysite.com/about.html
at 02:24:08 AM on Wednesday, September 13, 2023.
This visitor used Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/22.214.171.124 Safari/537.36.
Well that’s interesting. . . It’s almost like the bot bounced off CF with 126.96.36.199 and landed on my site with 188.8.131.52, effectively bypassing the CF challenge. wtf?
I have no idea how it’s doing this. The only thing that popped up on google was something called an ISP proxy where, apparently, bot masters can now access huge pools of good reputation IP’s and route requests through them, even from a server. The other possibility is a botnet consisting of many compromised drone machines I suppose. I just don’t understand how it is bypassing cloudflare.
Full firewall log re: 184.108.40.206
Sep 13, 2023 2:24:04 AM
Export event JSON
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/220.127.116.11 Safari/537.36
Empty query string
Another thing. . . This bot never changes its user agent. It always appears as:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/18.104.22.168 Safari/537.36.