Updated public Known Good Bots list

It would be good if the list here Known issues and FAQ · Cloudflare Firewall Rules docs was more up-to-date.

I have recently contacted support a couple of times regarding some good bots (updown.io and Alexa) and was told these already are in the cf.client.bot list but the documentation was not updated.

1 Like

SemRush bots are missing too in good bots.

I’ve griped about the outdated list before, as it bites me from time to time. Maybe @cwaters can convince someone to update it pretty-please.

1 Like

While I agree Semrush is a “Known bot” I wouldn’t class it as a “Known Good Bot”. I have lots of unsolicited scans coming from this bot - so much so that I have a firewall rule in place to block it altogether.

Some services should only scan sites after being authenticated (via DNS, file or header). Unsolicited scans from “SEO”, “Social monitoring” and others are of no benefit to my site and just consume CPU, memory, database space, log space and bring me… nothing really.

So there’s a difference between “Known bot” and “Known good bot” too.

3 Likes

The Known Bots should respect robots.txt
If they don’t, then I suggest you open a ticket and complain.

Seems QWANT is missing as a good bot, seems to be filtered.
AS199064 QWANT-AS