CloudFlare Still Lets in A HUGE Amount of Bot Traffic

Other than putting my wordpress site into “under attack mode”, I’ve pretty much enabled all of the security features possible within Cloudflare.

I’ve added a firewall against XMLRPC attacks with Cloudflare (good), and the Hotlinking blocker.

But otherwise, I had to installed a Proxy/VPN blocker on my wordpress today, and holy sh*t… so much bot traffic still getting through Cloudflare (just in the last 3 hours):

And no way am I doing custom firewalls for every single IPv4 on cloudflare.

So seriously, what’s going on with Cloudflare?

I’m not quite sure what you’re asking. You didn’t say which options you’re using, but it’s difficult to create generic settings that won’t create false positives for a vocal set of users. So they end up with a vocal set of users upset about false negatives. I’m sure they’ve weighed the pros and cons. What you’ve done (block VPNs and proxies) is not an acceptable solution for many site admins. Even at the Enterprise level, there’s no silver bullet. Just more fine-grained firewall rule options.

1 Like

So seriously, what’s going on with Cloudflare?

There is nothing wrong with Cloudflare, if we summarize and simplify the situation “you get what you pay for”.
A proper anti bot system is an extremely expensive field for which enterprises spend easily hundreds of thousands of dollars per month, and still can’t achieve a 100% hit ratio.
The reason it’s expensive is due to the ridiculous amount of research it requires as well as the computing power, when detecting more complex bots, you can take for granted that javascript and captchas wont be enough, you want to analyze their browser pattern as well as many other datasets, all in real time.

Cloudflare I believe has something similar for the enterprise customers, however, for the lower tier plans we have to rely on captcha challenges and deploying our own rules.

As for the VPN/Proxies issue, the fact is that you can’t accurately detect them with full precision, you have to take risks and guess who is really behind a vpn/proxy.
Absolutely nothing guarantees you that certain ip block that today is used for proxies wont be sold tomorrow to an ISP. There are services who claim to have a precise solution to detect proxies/vpn’s, from my experience, they leave many holes open and usually affect legit customers as well.

Cloudflare already receives lots of criticism from users due to “forcing” challenges such as the 5s or Captcha, making it more strict would only bring more people complaining about the service.

I hope that you understand that, no matter what you are told, no service will accurately block all these bots, they will sell their product with the fanciest words when in reality, the service is not as complex as they make it seem.
Specially not with a budget under $1000 per month, I have tried most if not all the WAF providers that are in the range of $20-500 and I always found myself coming back to Cloudflare because the differences between providers that are not enterprise are minimal.

Finally, I have to agree that the JavaScript challenge is not aging that well, the efficiency is dropping considerably now that many bots and botnets are implementing “browser emulation” onto their systems, a new challenge is required but I doubt we will see it anytime soon due to how expensive it is to develop a new system that will hold whats about to come in the next years.

2 Likes

we have also a large number of bots from France accessing our site earlier this week, could be coincidence.

I updated the robots.txt and .htaccess file (see this link https://support.acquia.com/hc/en-us/articles/360042181273-Block-Access-to-Bad-Bots-coming-from-the-Huawei-Cloud with a few adjustments) to stop bad bots, and now it seems to under control for now,

but the bot attack started this week, like yours,

So after a few more days of running the proxychecker/blocker, here are the results:

(I’ve highlighted the data from 1 day ago, refused queries is proxy/vpns that were no blocked because I ran over the 1000 queries/day)

The Proxychecker plan is free for 1000 queries, and only $3/month for 10,000 queries, so I don’t see how cloudflare does not have the ability to implement something similar to what they do with Argo Cost/Quota pricing model.

Cloudflare is already working on adding the ability to block (known) proxies.

In addition to Lists that you create and manage yourself, we plan to curate Lists that you can subscribe to and use in your rules. Our initial ideas revolve around surfacing intelligence gleaned from the 27M properties reverse proxying traffic through the Cloudflare edge, e.g., equipping you with lists of IPs that are known open proxies so requests from these can be treated differently.

3 Likes

That’s awesome. thank you arunesh. I’m sure there will be managed lists with proxy and vpn blockers.

Thanks for this.

This topic was automatically closed after 30 days. New replies are no longer allowed.