Error 403 for Search Engine Bots like Google Bot (Desktop) and Yandex

Hello Cloudflare Team,

My site ShoutMeBack is suffering from error 403 forbidden when search engine bots are trying to crawl my site. I even checked the server response with the Yandex server response checker, httpstatus.io, and with the Google Rich Result Test for Google Bot (Desktop Version). In both, I got error 403 for my site.

Currently tested:

  1. Deactivating all plugins
  2. Used default .htacess file
  3. Default robots.txt (no blocking found)
  4. Hosting Provider whitelisted all the Cloudflare IP ranges

I figured out that when the Yandex bot tries to access my site with Cloudflare IP address, it results in error 403 forbidden.

Here is a snapshot with the 403 Error response:

When I clear the Cloudflare cache and try again checking the response, it shows 200 OK response.

Here is a snapshot with 200 Ok response:

Do note that in 200 OK response, the IP address is 145.239.100.28 (That’s my servers IP address, not the Cloudflare IP). Hence, I get 200 OK for it.

Summary: The site responding with a 403 error with Cloudflare IP and with my server IP, the site is responding with 200 OK.

Please look into this problem ASAP because various other sites are also facing the same issue.

We don’t work for Cloudflare, but can offer some suggestions.

Check your Firewall page for the Event Log. If the 403s are coming from Cloudflare, they would usually show up there along with the reason the request was blocked.

Hey sdayman,

There are some block events like hotlink protection, security level and some other services. But I did not find the any search engine bots URL in the list.

However, I disabled all the firewall settings and tried but still the same result with error 403 on my site.

I noticed that the IP address that I am getting while checking response is of Cloudflare but they have not listed it in their Cloudflare IP ranges guide.

Can you please help me to solve the issue? It is really hurting the search engine rankings.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.