Blocked user-agent being wrongly whitelisted

firewall

#1

Hi,

Cloudflare Firewall for my website shows several visits under a “uablock” rule, i.e., user-agent blocking rule, that were nevertheless whitelisted instead of JS-challenged. This is a hacker attempting access to various php files, including: txt.php, system123.php, v4web.php and others.

I checked all my rules, and the only “Whitelist” action I have enbled are for Twitterbot, Facebook crawler and GTMetrix. The IP in case does not match any of those.

I have 4 Firewall Rules in place: Allow for known bots; JS challenge, challenge and block for 3 different cases. Also I have 8 UA and 4 IP rules in place with either Block or JS challenge.

The user-agent in case was supposed to be JS challenged either by one of my firewall rules based on country, or by the specific UA rule for this user-agent.

Any ideas of what could have gone wrong?


"Action Taken Whitelist"
#2

I know you’ve already checked a bunch of places, but it does seem that something applicable is whitelisted.

What does the Details link show?


#3

I do know that “uablock” is just short for the general “user-agent blocking”, while the action chosen could be block, challenge, JS challenge or whitelist. For this specific bot, it’s set at JS challenge.


#4

Yes, that’s what uablock is. And this is a good mystery. I bet there’s some obscure setting that’s doing this. Or a bug.

Maybe Support can track it down:
Login to Cloudflare and then contact Cloudflare Support


#5

Can you post a screenshot of your user-agent blocking rules?


#6

Sure:


#7

All right, just wanted to clarify if there could have been any contradicting rule

I’d take that 497fb00edebf68ba connection ID and forward it to Cloudflare support for them to check what happened. That does look like a potential bug.

@cloonan / @cscharff


#8

Thanks @sdayman and @sandro.

I’ll contact Support and update this post when I get their feedback.


#9

Just an update:

I haven’t yet heard from CF Support after their first, automated response.

I checked on my origin logs and yes, the visits were actually whitelisted and passed on to the origin (I was suspecting maybe there was a bug where the action "Whitelist’ was showing but the correct action, JS Challenge, was actually performed). The hits were blocked by my WordPress firewall.

The same UA came back today, probing the same URLs, was again whitelisted,


#10

For the time being I’d also add these user agents to firewall rules. Cloudflare’s rule engine(s) is/are quite powerful but, with all the recent changes, currently a bit of an unholy mess. These things should be hopefully sorted out in the coming months with the slow consolidation of these engines.

Refreshing the tags for @cloonan and @cscharff and, introducing, @alexcf :sunglasses:


#11

Could it be you have any other rule (not user agent, but access or firewall rule) set up which might interfere here?

I just set up a blocking test user agent rule and it seems to have worked. It got blocked and logged in the events list. So it seems it might not necessarily be a general issue but could be specific to your account configuration or - hence my earlier question - have something to do with other rules in your account.


#12

Hi @sandro,

I’ve had user agent rules for a while now (over a year if I remember it well) with great success. My Firewall events log shows that my rules are being followed properly. I have Access rules to protect /wp-login.php, /wp-admin, and /xmlrpc.php, all for both naked and www. domain. I have a few other Access rules for various subdirectories, but they have no connection with the visits we are talking about.

I have the following firewall rules in place now:

Firewall Rules (4):

  1. Know bots > Allow
  2. Visits not from selected countries > JS Challenge
  3. Visits to several WordPress URIs hackers normally try > Challenge
  4. Visits to other URIs hackers normally try > Block

I also have the following IP Access Rules:
4 rules with unique IPs > JS Challenge
Facebook AS number > Whitelist
Twitterbot AS number > Whitelist
GTMetrix (3 IPs) > Whitelist

A also have 8 User Agent blocking rules: (screenshot provided earlier on this thread)
3 with action JS Challenge
5 with action Block

I don’t see how any of these rules could have resulted in a unique IP address being whitelisted, but I’m open to consider changes if any of these rules can be reasonably pointed as the culprit.


#13

I wouldnt have thought so either, but I wanted to rule out any possible interference.

From the rules you posted only the “knowns bots” and the three access rules should allow anything, but I wouldnt assume either applied to the request in question.
I checked to whom that IP address belongs and it turned out to be OVH, so I guess we can rule out your “known bots” rule. We can probably equally rule out the Facebook and Twitter AS numbers and I really dont think you whitelisted the IP in question as “GTMetrix”.

Based on that I’d pretty confident too to rule out a rule interference of that type, which brings us back to something that might be either specific to your account configuration in the backend or a simple bug. So we are back to waiting for support :slight_smile:


#14

I just did a search on the Firewall Events log for that IP number, and it turned out the IP was repeatedly JS challenged before it was whitelisted, further confirming the rules in place should have stopped it.

These are hits from the same IP, same country, same User Agent, and to the same few URLs.


#15

Can you post the ticket # so @cloonan or @cscharff can look at it?


#16

Hi @sdayman,

Request #1624598 (I assume this is the ticket number, right?)


#17

Hi @floripare, yes. Thank you. I see the ticket, linked it to this conversation, will keep an eye on it.


#18

Whitelist was removed from User-Agent blocking as it doesn’t actually do anything. I have no idea why this was added, however you will most likely see that when creating new rules that this will not be an available option.


#19

Yeah, but they never whitelisted anything. Post #6 shows rules with only Challenge or Block.


#20

OK! Got to the bottom of this.

See here:
CloudApp

Scenario I tested to produce:

  • Create a UA Blocking rule
  • Visit site, receive captcha
  • Refresh a bunch of times (getting challenge logs)
  • Complete captcha (then get whitelist)

So basically, your bad guy is passing the Captcha! Change from using jsChallenge to using Captcha for your Firewall Rules.