Hello Cloudflare Team,
My site ShoutMeBack is suffering from error 403 forbidden when search engine bots are trying to crawl my site. I even checked the server response with the Yandex server response checker, httpstatus.io, and with the Google Rich Result Test for Google Bot (Desktop Version). In both, I got error 403 for my site.
- Deactivating all plugins
- Used default .htacess file
- Default robots.txt (no blocking found)
- Hosting Provider whitelisted all the Cloudflare IP ranges
I figured out that when the Yandex bot tries to access my site with Cloudflare IP address, it results in error 403 forbidden.
Here is a snapshot with the 403 Error response:
When I clear the Cloudflare cache and try again checking the response, it shows 200 OK response.
Here is a snapshot with 200 Ok response:
Do note that in 200 OK response, the IP address is 126.96.36.199 (That’s my servers IP address, not the Cloudflare IP). Hence, I get 200 OK for it.
Summary: The site responding with a 403 error with Cloudflare IP and with my server IP, the site is responding with 200 OK.
Please look into this problem ASAP because various other sites are also facing the same issue.