What is the name of the domain?
What is the issue you’re encountering
Google bots can not crawl the site
What steps have you taken to resolve the issue?
We have added googlebot useragent to the CF allowlist
What are the steps to reproduce the issue?
When trying to crawl the desktop Google bot on the homepage, we get a 4xx error. Also, when we make a request to the homepage with the Google-Bot Inspection Tool, we encounter a 4xx error. What causes this problem? And how can we solve this problem?
We do not experience such a problem on Googlebot Smartphone. Also, the problem we experienced with the Google bot desktop and Google-Bot Inspection Tool user agents above only occurs on our homepage. There is no problem when Google bot desktop and Google-Bot Inspection Tool crawl our other pages. These two user agents encounter a 4xx error only when crawling our homepage.
Note: We checked our blocked IP addresses, we could not find a Google bot IP. Also, we do not have any security layer to block Google bot. In addition, the Google bot user agent is already on our allowlist. In other words, we do not do anything to block Google bot on the hosting side or in our cloudflare settings. However, we experience such a problem on the homepage and we think the problem is caused by Cloudflare. Please help us solve this problem!