Documentation clarity around verified bots / WAF rules

Describe the issue you are having:
Hey team, just looking at the WAF menu under Security. I have a custom rule to block AI crawlers and it uses the expression (cf.verified_bot_category in {"AI Crawler"}), however the documentation over @ seems to be a bit out of date as bots aren’t sorted into an AI Crawler category but rather Other.

A good example is GPTBot, which is classed as “Other”

Is the table data accurate at the moment? As lodging queries through ChatGPT/Gemini/etc seemed to pass right through the WAF rule and the selected criteria above until I created an additional rule where I manually defined user agents and successfully blocked their requests.

Additionally, is there a publicly available list of blocked bot user agents? I’m keen to see if a few I know of which disrespect the robots.txt file are listed there, and if not whether I could submit them (bytespider for example)

Hi @BiosPlus, welcome to the Cloudflare Community.

I am checking with the team on this one for you.

I have raised this discrepancy with the Radar team earlier.

We tested from our end and the rule appeared to work for the requests it was intended to cover.

What was the User-Agent/IP data for those requests that are coming through? And please ensure that there are no IP Access Rules or other Custom Rules allowing to bypass security features.

There is not.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.