My website has been seeing a very high magnitude of “verified bot” traffic, which has increased substantially over the last several days.
At this point, 98% of my web traffic of over 750,000 requests in the last 24 hours, is coming from “verified bot” traffic. It seems like either there is a mistake, these are not “good” bots, or something is wrong with my configuration that I am not aware of.
Can someone let me know if they’ve seen something similar to this, and what if anything they have done to control their bot traffic?
I do already have firewall rules enabled, rate limiter enabled, and bot fight mode enabled.
Can you give an example of firewall rule you have in place? Do you know which known bots are being allowed (you could create an ALLOW rule for Known Bots and put at the bottom so it at least records this in the firewall logs).
I had only two rules in place prior to your recommendation. One was to block empty user agents, and the other was to challenge moderate threat scores. I’ve added the “Known Bots” rule and set it to allow the bot, but log the access.
So far I am seeing expected health checks from Cloudflare and another vendor. I’ll monitor this rule for unexpected behavior over the next week to see what it reports.
The number of requests to the domain has also fallen back to a “normal” level, from the 750k requests that were occurring at the time of this question’s posting.
