I recently noticed that my website technobird.com got completely eliminated from Google search results.
I tried URL Inspection tool in Google Search Console, it was throwing a General HTTP error (without much details). Even though I was able to open the website in browser without any issue.
I contacted Google Search team about the issue and here’s the response:
Thanks for reaching out. While your site can be accessed via a browser, Googlebot has a different behavior. This could be due to a protection system that’s blocking unknown requests to your site.
To learn more about Googlebot and other Google crawler Verification, check this article. Please reach out to your hosting service provider to check what’s the underlying issue and fix it to avoid further access issues.
I also opened a ticket with Cloudflare in which I explicitly mentioned that the website is working fine when I open it in browser and the error only occurs when Google bot tries to crawl the website. So apparently its Cloudflare blocking requests from googlebot
And the response I received was a canned reply asking for irrelevant things which clearly indicates that the support rep didn’t even bother to read the information mentioned in the ticket.
Thank you for contacting Cloudflare Support. We’re sorry to read that you’re experiencing difficulties.
In order to better assist you with the problem you are experiencing, we will need some additional information from you.
Can you please share the following with us:
- The specific error messages being returned and/or behaviours where you are seeing issues while on the website.
- Specific step by step instructions on how to reproduce on our end - e.g. if this issue is only replicable behind a login, can you provide a temporary test account for us
- A screenshot of the errors you are seeing.
- Any relevant access logs from your web server.
- A HAR file demonstrating the issue.
Please respond with that information as soon as you can so we can continue to work with you to resolve your issue.
- Cloudflare Error Messages (and what they mean)
- How do I check my server’s response directly without Cloudflare?
- Reporting a bug
Kay Lee | Cloudflare Support Engineer
Can anyone please guide how to have it fixed?