After install cloudflare i experience a 403 from pagespeed , search console(for example impossible to fetch robots txt) and we have lose 61% of traffic.
There’s some setting i miss to setup ? I see that we have
403 error on all pages from Google.
Please it’s important i don’t know what’s happen.
Regards from Enrico.
May I suggest below article for more information about this error:
Try the suggestions in this Community Tip to help you fix Error 403 Forbidden.
403 Forbidden Error is a client side error that means that the client sent something the origin was unable to process. With the exception of requests that violate WAF rules or subdomains that are not covered by a certificate, Cloudflare does not generate any 4xx errors, so this would indicate something is not configured correctly with your hosting provider or your client is sending something inco…
In your robots.txt file I do not see any entry which could allow Google to fetch your sitemap.
Either define again (duplicate) User-Agent: * and Sitemap: path or define manually User-Agent: Googlebot (multiple).
Otherwise, do you see any blocked or challenged events at
Firewall tab → Firewall Events from Cloudflare dashboard for your domain name?
Are you using any custom-made Firewall rules like country blocking, or maybe Bot Fight mode is enabled, etc. at Cloudflare dashboard?
You do have some syntax error, duplicate “content” attribute?:
<meta name="robots" content="index,follow" content="max-image-preview:large" />
<meta name="robots" content="index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1">
Bot fighting is enable. Could be this ?
Which Bot Fight? Regular, or did you configure Super Bot Fight Mode?
Regular. Anyway from the day i set cloudflare this problem happen before no problem.
Around 6k bot attack in 30 minute for that i switch on.
It wouldn’t surprise a lot (just a little) if regular Bot Fight Mode blocked Pagespeed Insights because it’s supposed to let Google’s crawler through. I guess Pagespeed Insights isn’t on the same approved list as the Google crawlers.
I think it was heavy firewall of cloudflare set to high on both DDoS and firewall.
Google it’s a bot so maybe cloudflare block all bots because after change setting to medium it start to crawler again.
Anyway cloudflare was the only change before that happen, so I don’t know why but stop Google bot and give
Hope to have solve in this way because before everything working good.
I update till 2-3 days to tell that it’s true or not.
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.