I am facing this problem after I changed the server of my site, so that when I changed the DNS (from Bluehost to cloudflare) and when I publish a new article on the site and submit it in the Google console to be indexed, I see the message (URL is not available to Google
, Page cannot be indexed: Server error (5xx)), knowing that the article has been published and I can visit it from different devices from the main page of the website!!! Has anyone ever faced this problem and what is the solution in your opinion?
Thanks for contacting Cloudflare Community, sorry to read that you’re experiencing difficulties.
Can you please try the following procedure:
Visit Google Search Console, use the “Inspect URL” tool, then click on the “Test Live URL” button.
After Google finishes fetching the page, if the result is that it was blocked by a 403, head over to Cloudflare Dashboard > Security > Events.
Here you will find the event related to this URL block. It will give you details on what Cloudflare service has actually blocked/challenged the request, then you can adjust your settings accordingly, by either editing a WAF Custom Rule to avoid blocking Known Bots, or disabling/skipping services such as (Super) Bot Fight Mode.
If you cannot find a related Security Event, Googlebot is likely being blocked by the origin server, so I would suggest checking your server firewall logs.
I hope this information was useful, let us know if we can further assist you in this matter.
Check out this link for a step-by-step guide on how to fix the Googlebot Blocked by Cloudflare: https://wpbasicpro.com/how-to-fix-googlebot-blocked-by-cloudflare-without-compromising-security/