403
When cloudlfare proxy is enabled, it returns 403 error for every url. Right now I disabled it to work around the issue.
Disable Cloudlfare Proxy
Enabled Cloudflare Proxy
Enter any url from our site yellowchilis.com in https://httpstatus.io/ and check for the http response.
it is difficult to diagnose when cf is disabled, works for me, but suspect it’s still disabled?
Yes, it was disabled. But I enabled the proxy now. Can you check it out?
Loads for me fine, can you clear cache and/or try incognito mode? I am using chrome on a mac
It works for me too. I tested it on this site https://httpstatus.io/ and it does indeed return error 403. It’s probably some rule in your firewall that’s blocking the HTTPStatus bot or Bot Fight Mode is blocking it. Check the logs and analytics on your account.
Cloudflare didn’t block it when I set it to use the DuckDuckGo bot or when I set the user agent to httpstatus/2.0.
But why are you using this tool?
@cloonan It loads but returns 403 for most of the search engine bots including Google. That’s the issue here.
@WhiteDemonhia Expect DuckDuckGo, for every other bot it responds 403. I used this tool to diagnose.
If you see Google Search Console data below, it reports 4XX response code for all of the urls it crawls. This directly impacts the indexing and ranking of the entire site.
I see. Can you please go to your analytics and WAF and see why they are being blocked? 
To do this, send about to 5 requests to your site (from google/bing bots or from httpstatus). Then open two tabs, one for analytics and one for events. Depending on the level of your account, it may take up to 5 minutes for the events to appear. So, if nothing appears, wait 2 minutes and refresh the pages until the events show up. Click on events and it will tell you what triggered the firewall.
If after this you need more assistance, send us screenshots so that the community can help you better. 
Thank you for your response. I did follow your instructions and here is what I got. It seems the managed rules trigger 403. I had the old cloudlflare managed rules deployed, I upgraded and deployed new managed rules. Also, tried disabling the rules all together like below. But nothing seems to fix 403 response even for google bots.
The system doesn’t allow me to attach more media in one reply so here are additional snapshot.
Analytics:
^
You can see that Bot Fight Mode for Definite Bots is the culprit.
Though you can also see that it is a fake GoogleBot, so it was blocked for good reason.
As @Laudian mentioned, it was not Cloudflare Managed Ruleset that triggered the firewall, it was Bot Fight Mode.
If Bot Fight Mode is blocking the real Google bot and other search engines, you need to take a look at the bot settings.
Set Definitely Automated to Block if you want to continue using Super Bot Fight Mode.
Under Verified Bots, set the option to Allow.
In the Wordpress option, it will depend a lot on your case. Even on my Wordpress sites I didn’t need to check the Optimize For WordPress option. So if you have Wordpress, do an extensive test and then decide if it’s possible to leave it disabled.
In Javascript Detections, leave it enabled, as this helps Bot Fight Mode detect a malicious bot.
Static Resources: For me, works fine with it activated. But test yourself.
.Right, now that you’ve done that, send a bot from Google, Bing or another search engine to crawl your site. Send the real bot, not the one from third-party sites. Go to your webmaster and ask the real bot to scan your site.
It can take hours or even days for the Google/etc bot to crawl your site again.
If the real bot continues to be blocked, add a rule to your Firewall to skip known bots.
Whenever I need to use a skip rule, I always like to mark the requests to generate logs.
After that, set the rule position to Last and save. Now wait 30 seconds and try to send a new request for the real bots to crawl your website. This can take another set of hours or days for the bot to re-crawl. So, be patient. 
And Always keep checking the WAF Analytics and Events.
A more aggressive rule would be:
(cf.client.bot) or (http.user_agent contains “Google”) or (http.user_agent contains “Bing”) or (http.user_agent contains “google”) or (http.user_agent contains “bing”)
And just mark to skip the Super Bot Fight Mode and save it with the order Last.
And make sure to activate 
Cloudflare Managed Rules. Because when theses rules are active, Cloudflare will block fake Google and Bing bots, if a malicious bot using Google/Bing user agent bypass Super Bot Fight Mode. I don’t recommend to use the rule to skip user agent Google/Bing without Cloudflare Managed Rules active, because like I said, it would allow malicious actors to bypass your firewall by spoofing their user agents.
I hope this solve your problem. Cheers!
@WhiteDemonhia Thank you so much for your detailed response. I will follow the steps and see how it goes. @Laudian Thank you as well.
You’re welcome! 
This topic was automatically closed after 15 days. New replies are no longer allowed.
