Evidence of bot bypassing all of cloudflare challenges

I know this is a constant cat and mouse game but we have a very obvious web scraper hitting us right now that is bypassing all challenges being presented by our rate limiting rule (having already gotten around the managed challenge we present to all visitors on this particular route).

Is there a way to flag this or otherwise highlight the traffic in Cloudflare so you guys can update your challenge screens?

I’ve had to just have the rate limiter block access for now, which isn’t great.

Hello,

What is the zone in question that the bots are bypassing the “managed challenge”? What is the user agent string of this specific bot?

Also, how are you determining that this bot is bypassing us? Is this via access logs?

Any information on the zone and the User agent string itself that is bypassing us will greatly help us test this out. Also, if you can post a copy of your rule you made to catch this bot. That allows us to see what suggestions we can make for it.

2 Likes

I’d rather not share the zone publicly but if there’s a way I can send a DM or contact via email I can share that way.

Just a little background, we’ve had issues with scrapers for a while and Cloudflare’s anti-bot system has handled them pretty well up until now. We’re on the business plan and have super bot fight mode configured to block definitely automated, challenge likely automated, and allow verified bots for the entire zone.

In addition to this, we have a WAF rule to challenge all traffic except for known bots on a particular path that is the main target of the scrapers.

Over the past weekend we noticed a spike in traffic on this path and then checking Cloudflare logs identified a specific user-agent and ASN making the requests. Its pretty easy to tell when we’re getting scraped as they always enumerate through the resources on that path in a specific order, and the requests are coming in several dozen in a minute which is much more frequent than a normal visitor.

The user-agent is Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 and the ASN is T-MOBILE-AS21928. I can get the exact IPs if needed but they all begin with 172.56..

As I mentioned at the moment we’re mitigating the traffic via a rate limiter targeting that user-agent.

Also not sure how to post a copy of a WAF rule, do you just mean a screenshot of it? I don’t see a way to export/copy the rule as JSON or something like that.

Just following up here - should I open a ticket with support?

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.