We have an ongoing issue with bots that are not attacking, but interacting with our website. It’s not DDOS nor are they trying injections or anything else. Essentially an IP hits the site, loads a page, and then clicks on ads, for a couple of dozen pages on one day, then a hundred the next (sometimes a third day, but no more), and then we never see that IP address again. This will happen from 10 or 15 IPs on a daily basis. The only issue is that it messes up our ad counts. Never really a performance issue and there have not been any site attack issues.
I’ve set up Cloudflare in a trial with one of our sites with the free account (we have 16 news portal websites like this). We only see this traffic pattern on a couple of sites. I turned on the Bot Fight Mode and indeed, the report shows that it is identifying the IPs (clear to see in the report) and it marks them Managed Challenge, but then lets the bot through (apparently).
How can we configure this to actually block these identified bots? Happy to set up a subscription, just don’t know which version to set up. Which of the subscriptions can convert these VPN-based “Managed Challenge” visitors?
Not exactly an answer to the question “How can we configure this to actually block these identified bots?” but to “It’s not DDOS nor are they trying injections or anything else.”
It can actually be a DDOS, where instead of taking down your server the attacker could be simply trying to cause problems with your ad accounts to the point where companies won’t deal with you. This can starve your business of revenue, causing it to ultimately close or scale down operations considerably.
Assume these are attacks if the consequences of these actions could compromise your ability to continue in business and put these in your business continuity plan.
That’s not a DDoS; it’s a targeted bot attack with click fraud behavior.
@user68510 You need a bot management solution; your best bet would be Cloudflare enterprise. I doubt you will achieve much in any of the lower-end packages due to the nature of the attack.
Challenge won’t work. This kind of attack is typically blocked without the chance of allowing malicious traffic to go through. Let me emphasize. No challenge will work against these bots, not even CAPTCHA.
You can also choose another enterprise-grade bot management; however, rest assured that these solutions aren’t cheap. If you are going to spend money on a bot management solution, you should expect to spend no less than $20k per annum.
Sure, its possible, but its not a big company with millions of followers. I see no advantage that would be derived by pushing this company out of business. There’s very little in the market…its very niche…
That said, I do believe that Cloudflare could resolve this, unfortunately, the Enterprise version at $5000/site/month ($60k/year!) is out of the reach of all but the fortune 2000. That’s an entire FTE! $20k is better, but still a reach for the SMB market…who would have a solution for $20k or less?
Setting up and adjusting daily firewall rules in cloudflare (great threads in search about ideas, heaps of ideas), getting logs sent daily from cron jobs, investigating seeing new trends, watching google analaytics for spikes or better yet hosted analytics to grab better info.
Then you can create on your actual origin server honeypots, nginx is great for this easy to set up, can give error 5xx and then ban them with fail to ban is what I do, set up whitelist black lists, cron jobs the lot, most can be automated.
It aint enterprise bot management, but you can sure wreck most idiotic bot operators this way.
Sadly, if you cannot afford to pay, you will have to afford time into fighting this never ending battle.
Many vendors have solutions for $20k or less. However, it comes at the expense of not being as good. Unfortunately, the bots you describe are pretty hard to detect and mitigate. There are exceptions, and it depends entirely on the bots attacking your company.
Just a heads up, there is one vendor in particular that I believe goes for $16k (starting price), do not buy it; IMHO, it’s among the worst solutions in the market as they rely on CAPTCHA and IP reputation heavily.
I believe that Shape security might be able to help you out. However, I’d also try and schedule a meeting with a CF representative as maybe they can help you out if you are only interested in bot protection.
These bots require fingerprinting and header validation among other deep inspection techniques to be detected, firewall rules won’t be of use in this case, unfortunately.
I tried to have the pricing conversation with someone who answered the sales phone. They were frankly, singularly direct, and unmoving. Only the Enterprise version at $5000/month/site works for this issue. $60k/year is an FTE for an SMB. That number doesn’t work. It’s interesting, CF is not rated very well by Forrester (https://reprints.forrester.com/#/assets/2/1782/RES158095/reports, you’d think they’d be after marketshare at a competitive price. Guess not…
I see where you are coming from, and I understand that it’s expensive; however, PerimeterX, for example, starts at $100k per annum. Akamai typically starts at $500k and over; Imperva is also quite expensive.
Bot management is a costly task; you need the best engineers in the field and a lot of computer power; cheap solutions typically rely on client-side security and little to no server-sided checks.
Given what you said + that Bot Fight mode eventually let’s the user through after a challenge, that sounds like it could just be human users doing this rather than automated bots? In such case even Bot Management Enterprise edition might not help though.
It is capable of detecting those. Most click fraud bots do not accurately emulate the user’s click or movement. However, no protection is a silver bullet, and there is room for bots going through the protection.
The best solution IMHO is TLS or canvas fingerprinting; both are very hard to bypass
.@user68510 I believe that you could still use CF Pro or business along with a webserver such as enterprise haproxy. You could use Haproxy fingerprint module to detect bots and then ban them at the edge using CF.
Note, this haproxy layer would need to be exposed in order to keep track of the original TLS fingerprint.
This is likely the best solution if you are on a budget, however, it requires some system administration tunning and expertise in the field.
Finally, I have not tested haproxy enterprise myself but since they have a TLS fingerprint module, it should do exactly what you need in this case.
The best option would be to ask them for a trial and verify it yourself.
Interesting I am using Enterprise Bot Management and it’s blocking between 3 to 8 million bots per week majority from my own firewall rules. But I haven’t really been subject to ad click fraud that I know of
Hey I get the point, however, the more important point is that the SMB market cannot afford those kinds of numbers. There has to be a solution that can fit into the budget of those without infinite budgets and IT teams…
The point I see is that this is certainly not human activity (no one clicks 100 times in a second navigating across all ads on the site) and according to the reports, its marking these VPN Based attacks as ‘managed challenge’. Not sure what that means, but its identifying them correctly. Perhaps Super Bot Fight Mode can do that?
Unfortunately not, what’s evident to us requires real-time monitoring and prevention. Proper bot management will diagnose in real-time and deliver an estimation of bot store the moment they connect to your site.
Again, I’m not saying there aren’t bot management that costs $50-$200 a month. My point is that they are weak and easy to bypass. You could gamble it and see if those work out; however, I can guarantee that they won’t be a reliable solution in the mid-long term.
If this is the most obvious behavior, you can mitigate it partially with a rate limit
Doubtfully, unfortunately, SBFM is unreliable as it causes a lot of stability issues.