Block AI Scrapers and Crawlers

General
1

What is the name of the domain?

https: // facts institute .com

What is the issue you’re encountering

potential conflict

What steps have you taken to resolve the issue?

I have a plugin called Robots.txt optimization and want to make sure its not in conflict with “Block AI Scrapers and Crawlers” setting on cloudflare. I have the following settings turned on in Robots . txt optimization: Block Bad Bots, Block Xenu, Block Majestic Crawler, Block Moz Crawler, Block Semrush Crawler, Block Ahrefs Crawler, Block WayBackMachine Archive.org Crawler. Should I turn them off and just leave Cloudflare’s “Block AI Scrapers and Crawlers” running? Thank you!

2

Hi there caitrionamctiernan

At this situation I would advise to leave Cloudflare’s “Block AI Scrapers and Crawlers” running and if it isn’t enough you can specify within a WAF rule a block action on those specific bots and of course to take into consideration the list here:

https://radar.cloudflare.com/traffic/verified-bots

If opting to add the WAF rule be attentive to have “Verified Bot Category”

Hope this clarifies.

Cheers!

3

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.