Finding the right Bot Management Approach

We have an ongoing issue with bots that are not attacking, but interacting with our website. It’s not DDOS nor are they trying injections or anything else. Essentially an IP hits the site, loads a page, and then clicks on ads, for a couple of dozen pages on one day, then a hundred the next (sometimes a third day, but no more), and then we never see that IP address again. This will happen from 10 or 15 IPs on a daily basis. The only issue is that it messes up our ad counts. Never really a performance issue and there have not been any site attack issues.

I’ve set up Cloudflare in a trial with one of our sites with the free account (we have 16 news portal websites like this). We only see this traffic pattern on a couple of sites. I turned on the Bot Fight Mode and indeed, the report shows that it is identifying the IPs (clear to see in the report) and it marks them Managed Challenge, but then lets the bot through (apparently).

How can we configure this to actually block these identified bots? Happy to set up a subscription, just don’t know which version to set up. Which of the subscriptions can convert these VPN-based “Managed Challenge” visitors?

Not exactly an answer to the question “How can we configure this to actually block these identified bots?” but to “It’s not DDOS nor are they trying injections or anything else.”

It can actually be a DDOS, where instead of taking down your server the attacker could be simply trying to cause problems with your ad accounts to the point where companies won’t deal with you. This can starve your business of revenue, causing it to ultimately close or scale down operations considerably.

Assume these are attacks if the consequences of these actions could compromise your ability to continue in business and put these in your business continuity plan.

That’s not a DDoS; it’s a targeted bot attack with click fraud behavior.

@user68510 You need a bot management solution; your best bet would be Cloudflare enterprise. I doubt you will achieve much in any of the lower-end packages due to the nature of the attack.
Challenge won’t work. This kind of attack is typically blocked without the chance of allowing malicious traffic to go through. Let me emphasize. No challenge will work against these bots, not even CAPTCHA.

You can also choose another enterprise-grade bot management; however, rest assured that these solutions aren’t cheap. If you are going to spend money on a bot management solution, you should expect to spend no less than $20k per annum.

It is a Denial of Service attack as in the end result is the OP being unable to service visitors if the business goes under. The click fraud behaviour is the method they achieve the DDOS.

Sure, its possible, but its not a big company with millions of followers. I see no advantage that would be derived by pushing this company out of business. There’s very little in the market…its very niche…

That said, I do believe that Cloudflare could resolve this, unfortunately, the Enterprise version at $5000/site/month ($60k/year!) is out of the reach of all but the fortune 2000. That’s an entire FTE! $20k is better, but still a reach for the SMB market…who would have a solution for $20k or less?

Daily fighting

Setting up and adjusting daily firewall rules in cloudflare (great threads in search about ideas, heaps of ideas), getting logs sent daily from cron jobs, investigating seeing new trends, watching google analaytics for spikes or better yet hosted analytics to grab better info.

Then you can create on your actual origin server honeypots, nginx is great for this easy to set up, can give error 5xx and then ban them with fail to ban is what I do, set up whitelist black lists, cron jobs the lot, most can be automated.

It aint enterprise bot management, but you can sure wreck most idiotic bot operators this way.

Sadly, if you cannot afford to pay, you will have to afford time into fighting this never ending battle.

It sucks sadly but thats net life these days.

Many vendors have solutions for $20k or less. However, it comes at the expense of not being as good. Unfortunately, the bots you describe are pretty hard to detect and mitigate. There are exceptions, and it depends entirely on the bots attacking your company.

Just a heads up, there is one vendor in particular that I believe goes for $16k (starting price), do not buy it; IMHO, it’s among the worst solutions in the market as they rely on CAPTCHA and IP reputation heavily.

I believe that Shape security might be able to help you out. However, I’d also try and schedule a meeting with a CF representative as maybe they can help you out if you are only interested in bot protection.

These bots require fingerprinting and header validation among other deep inspection techniques to be detected, firewall rules won’t be of use in this case, unfortunately.

1 Like

I tried to have the pricing conversation with someone who answered the sales phone. They were frankly, singularly direct, and unmoving. Only the Enterprise version at $5000/month/site works for this issue. $60k/year is an FTE for an SMB. That number doesn’t work. It’s interesting, CF is not rated very well by Forrester (https://reprints.forrester.com/#/assets/2/1782/RES158095/reports, you’d think they’d be after marketshare at a competitive price. Guess not…

I see where you are coming from, and I understand that it’s expensive; however, PerimeterX, for example, starts at $100k per annum. Akamai typically starts at $500k and over; Imperva is also quite expensive.

Bot management is a costly task; you need the best engineers in the field and a lot of computer power; cheap solutions typically rely on client-side security and little to no server-sided checks.

1 Like

Given what you said + that Bot Fight mode eventually let’s the user through after a challenge, that sounds like it could just be human users doing this rather than automated bots? In such case even Bot Management Enterprise edition might not help though.

It is capable of detecting those. Most click fraud bots do not accurately emulate the user’s click or movement. However, no protection is a silver bullet, and there is room for bots going through the protection.

The best solution IMHO is TLS or canvas fingerprinting; both are very hard to bypass

.@user68510 I believe that you could still use CF Pro or business along with a webserver such as enterprise haproxy. You could use Haproxy fingerprint module to detect bots and then ban them at the edge using CF.
Note, this haproxy layer would need to be exposed in order to keep track of the original TLS fingerprint.
This is likely the best solution if you are on a budget, however, it requires some system administration tunning and expertise in the field.

Finally, I have not tested haproxy enterprise myself but since they have a TLS fingerprint module, it should do exactly what you need in this case.
The best option would be to ask them for a trial and verify it yourself.

1 Like

Interesting I am using Enterprise Bot Management and it’s blocking between 3 to 8 million bots per week majority from my own firewall rules. But I haven’t really been subject to ad click fraud that I know of

2 Likes

Hey I get the point, however, the more important point is that the SMB market cannot afford those kinds of numbers. There has to be a solution that can fit into the budget of those without infinite budgets and IT teams…

~WRD000.jpg

Its not human. They do all their navigation and clicking in less than 1 second.

~WRD000.jpg

The point I see is that this is certainly not human activity (no one clicks 100 times in a second navigating across all ads on the site) and according to the reports, its marking these VPN Based attacks as ‘managed challenge’. Not sure what that means, but its identifying them correctly. Perhaps Super Bot Fight Mode can do that?

~WRD000.jpg

In that case, then only solution I see is Enterprise Bot Management to more finely target who is human vs bot.

Example

2 Likes

Unfortunately not, what’s evident to us requires real-time monitoring and prevention. Proper bot management will diagnose in real-time and deliver an estimation of bot store the moment they connect to your site.

Again, I’m not saying there aren’t bot management that costs $50-$200 a month. My point is that they are weak and easy to bypass. You could gamble it and see if those work out; however, I can guarantee that they won’t be a reliable solution in the mid-long term.

If this is the most obvious behavior, you can mitigate it partially with a rate limit

Doubtfully, unfortunately, SBFM is unreliable as it causes a lot of stability issues.

3 Likes
  1. If you see a non-human behaviour like 100 clicks a second, a client side solution might work, at least partially.

  2. I’d try to start with one of the cheaper solutions such as https://fingerprintjs.com
    Perhaps it will block enough of that traffic to the point that it’s not a problem anymore

  3. There is also this service Stop wasting your PPC budget on bot clicks | ClickCease™ which seems to be very specific to the problem that you talk about. I’m not sure if they solve ad-bots for ads running on your site, or just for your own ads, but could be worth talking to them.

1 Like

Very helpful, thanks!

Here is something you could do if you have a dynamic website and developers on hand…

  1. Add Google reCaptcha v3 to your pages reCAPTCHA v3 | Google Developers
  2. Fingerprint the client FingerprintJS Pro - Browser fingerprinting and fraud detection API
  3. Submit browser fingerprint and reCaptcha score to your server via xmlhttp for each page visit and store in your database with a timestamp.
  4. On your server calculate an average for that browser fingerprint over a period - the last three to seven days of data
  5. Make ads conditional on your page template. If the score is below a threshold for example 5 then set a flag and do not inject the ads on your page.

The result is suspicious traffic won’t see ads and it can be blocked from seeing ads in as few as three pages.

You can expand this by setting a cookie and then create a Cloudflare firewall rule that if that cookie is present with a certain value just BLOCK the traffic.

Yes, they could block scripts, reCaptcha, fingerprint but if they are malicious and want to click ads they won’t be blocking scripts at least and not Google properties so reCaptcha should work.

Use this with a list of known ASNs that have bad traffic (most cloud providers or colocation hosts should be blocked) and you will be halfway there. Remember to ALLOW Known Bots before blocking ASNs.

1 Like