ricardosdetailgarage.com.au
no error number
Sucuri sees the Cloudflare anti-bos as malware
Sucuri is being blocked from doing a monthly malware checking in our sites
turn Bot Fight Mode off
Turn Bot Fight Mode Off - run Sucuri scan
Turn Bot Fight Mode On - can’t run Sucuri scan
You can’t.
Bot Fight Mode doesn’t have such options for granularity, that you’re looking for.
See: Get started with Bot Fight Mode · Cloudflare bot solutions docs
OK thanks for the quick response.
Nutshell: paid CF you can, free CF you can’t.
Fair enough.
It is due to a technical limitation, where those two things are implemented in different stages, where it wouldn’t be possible to “override” it, if you insist on using Bot Fight Mode.
Whether Super Bot Fight Mode uses more resources, or if there are other specific reasons, explaining why it is (currently) only available on the paid plans, isn’t anything that I have a definite answer on.
You should however be able to accomplish your goal, but that will require a little bit of fiddling from your end.
Create your own WAF rules, that block traffic, that you don’t want.
Before these WAF rules, yuo’re creating a rule that Skip Verified Bots.
According to https://radar.cloudflare.com/bots#verified-bots, it seems like Sucuri is a Verified Bot at the moment:
Creating the WAF rules is a task that you have on your shoulders, … but you can do it with the kind of granularity that you are looking for.
That works also with the Free version of Cloudflare.
Well I thought that allowing “Known Bots” would resolve this since CF say it is aware of it. Or that I could add it in a WAF rule to whitelit specifically, but I don’t know what to be telling CF to skip.
It still won’t skip the Bot Fight Mode, - so that needs to be disabled anyway.
But you can skip the following WAF rules you have.
Some website owners want to block traffic from Workers, other people people wants to block complete ISP’s (e.g. AS numbers of hosting providers, or so), or even countries or continents.
Take the example in this screenshot:
The WAF rules are then doing the following:
I’m skipping Known Bots.
I’m blocking AS number 0 (typically when new IP ranges / long term unused IP ranges) appear on the Internet, they may be assigned to AS number 0 for a little while, until that correts itself.
I’m blocking Workers.
I’m blocking visitors, where the User-Agent string contains “bot”, but where the visitor is not a Verified Bot.
I’m blocking visitors, that aren’t presenting an User-Agent at all.
The well-known coding platform, GitHub, to mention an example, they have like forever been blocking access to their API interfaces, if no User-Agent was being presented to them by the client.
The #1 Skip Known Bots at Rule #1 in this example, is skipping Rule #2, #3, #4 and #5, so if it is a known / verified bot, it can technically come from AS number 0, or even without an User-Agent, according to this WAF (although, I doubt either of them will ever happen though).
You will need to figure out what traffic you believe is hostile, and then add patterns that matches that, as a “Block” rule, that is ordered after the rule that has “Skip” to “Known Bots”.
Well, we have used these as standard settings right across all of the sites that we manage. Experience shows that if we set any level of security above minimal, on some sites I can’t avoid using these as LE renewals seems to get stopped otherwise, and SEO-PowerSuite can’t reliably scrape sites that we manage. We only use these settings and light js challenges for other countries (since we only want mainly national traffic for most of our sites not international)
I also notice that CF has removed the manual security level setting and manage it differently now. I would have assumed that allowing known bots, as we do across all sites would ensure that Sucuri wasn’t blocked regardless of whether Bot Fight Mode was on or not - and yet here I am. Some of the sites that we run exactly the same scans on refuse to allow Sucuri to check for malware.
I appreciate your input, but I’m still in the same situation as before. I am yet to figure out what the differences are between the sites that would stop CF from allowing Sucuri through in some cases. Basically we have to turn Bot Fight Mode off to allow Sucuri to access those sites. Not the optimal setting.
Both Let’s Encrypt and Google Trust Services are at least verified bots, according to the list.
So if you’re setting up your WAF according to your wishes, then they will be allowed to pass through.
Bot Fight Mode has always been an “all or nothing”.
Therefore Bot Fight Mode is only advised as a temporary mitigation / last resort, against an attack from a swarm of bots on your website.
If you don’t have a such ongoing attack, - turn it off.
If you feel that way, as I said above, then Super Bot Fight Mode, on the paid plans, are (and have always been) the only possibility, in order to reach your “optimal setting”.
→
You can go the DIY (Do-It-Yourself) way just fine, on the Free plan, and accomplish your goal.
You can upgrade to the paid plan, with Super Bot Fight Mode, that can have do the granularity you want (such as allowing Verified Bots)
The “Bot Fight Mode” does NOT, and have never worked the way you want.
With things, such as e.g. this, it sounds more like that you’re being “deceived” by a WAF rule or so, where you are challenging traffic, that prevents Sucuri from accessing your site (e.g. no exception for Known / Verified Bots on that WAF rule).
During troubleshooting, you may eventually have enabled Bot Fight Mode, and caught that Bot Fight Mode was (now) blocking Sucuri, when what you were looking for before, was the WAF rule, that was challenging Sucuri.
OK, I use SpinupWP to manage our servers and they recommended the setup as I showed it, in cases where LE hasn’t been able to complete certification, which solves the issue.
But I see your point, and expect you’re quite right. It is quite possibly the JS challenge for overseas traffic which is problematic. I will disable that and see how this changes things. As you say, CF do say they allowlist Sucuri so it shouldn’t be an issue.
And I am hearing you regarding the Bot Fight Mode - to turn it off.
Thanks for your patience with this - I have learnt from this, and it has helped me a lot.
