Cloudflare is blocking robots from accessing robots.txt

I can see in the WAF log it’s being blocked by a browser integrity check, so I created a rule to allow it. Now I see the allow log entry followed by the block entry.

Does anyone know how to let robots access my robots.txt?

1 Like

It should be a Bypass instead of an Allow:

https://support.cloudflare.com/hc/en-us/articles/200170086-Understanding-the-Cloudflare-Browser-Integrity-Check

3 Likes

that worked, thanks!

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.