My website has been seeing a very high magnitude of “verified bot” traffic, which has increased substantially over the last several days.
At this point, 98% of my web traffic of over 750,000 requests in the last 24 hours, is coming from “verified bot” traffic. It seems like either there is a mistake, these are not “good” bots, or something is wrong with my configuration that I am not aware of.
Can someone let me know if they’ve seen something similar to this, and what if anything they have done to control their bot traffic?
I do already have firewall rules enabled, rate limiter enabled, and bot fight mode enabled.
Not seeing this distribution:
Can you give an example of firewall rule you have in place? Do you know which known bots are being allowed (you could create an ALLOW rule for Known Bots and put at the bottom so it at least records this in the firewall logs).
Hi @freitasm, thanks for your help.
I had only two rules in place prior to your recommendation. One was to block empty user agents, and the other was to challenge moderate threat scores. I’ve added the “Known Bots” rule and set it to allow the bot, but log the access.
So far I am seeing expected health checks from Cloudflare and another vendor. I’ll monitor this rule for unexpected behavior over the next week to see what it reports.
The number of requests to the domain has also fallen back to a “normal” level, from the 750k requests that were occurring at the time of this question’s posting.
The vast majority of these requests are still known bots, but they do not appear to be requesting resources at the same rates.
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.