Cloudflare blocking robots.txt

If you’re seeing a 403 error without Cloudflare branding, this is always returned directly from the origin web server, not Cloudflare, and is generally related to permission rules on your server. The top reasons for this error are: 1. Permission rules you have set or an error in the .htaccess rules you have set 2. Mod_security rules. 3. IP Deny rules Since Cloudflare can not access your server directly, please contact your hosting provider for assistance with resolving 403 errors and fixing rules. You should make sure that Cloudflare’s IPs aren’t being blocked.

Cloudflare will serve 403 responses if the request violated either a default WAF rule enabled for all orange-clouded Cloudflare domains or a WAF rule enabled for that particular zone. Read more at What does the Web Application Firewall do? Cloudflare will also serve a 403 Forbidden response for SSL connections to sub/domains that aren’t covered by any Cloudflare or uploaded SSL certificate.

If you’re seeing a 403 response that contains Cloudflare branding in the response body, this is the HTTP response code returned along with many of our security features:

  • Web Application Firewall challenge and block pages
  • Basic Protection level challenges
  • Most 1xxx Cloudflare error codes
  • The Browser Integrity Check
  • If you’re attempting to access a second level of subdomains (eg- *.*.example.com ) through Cloudflare using the Cloudflare-issued certificate, a HTTP 403 error will be seen in the browser as these host names are not present on the certificate.

If you have questions contact Cloudflare Support and include a screenshot of the message you see or copy all the text on the page into a support ticket.