Google is unable to Crawl site ,might be ips are blocked by Firwall

Google is not able to index this [site] and while connecting with google team they said "I understand that you’re facing issues with the indexing of your page Url Given Below, due to the fact that the system consider it “indexed without content” in the inspection tool. While some pages of your site can be accessed through a browser, Googlebot crawler is heading to a wall. This issue could be because the connection is being denied in one of these 3 phases: The CDN or Cache server A traffic management system or firewall The web server itself Usually, the issue happens in the second phase. A system might be blocking the access from an IP that it doesn’t recognize, and by chance it’s Googlebot trying to connect. Learn how to debug these network problems. To determine if the firewall is blocking, check if a connection is coming from Googlebot. It will do a reverse DNS check. To check again if it points to the correct IP, use the result. You can also ask Google to recrawl your urls. When you request a crawl, it’s not guaranteed that inclusion in search results will happen instantly or at all, even if a page meets all requirements. If you’re not sure how to fix issues with the steps listed above, contact your hosting provider with the information provided in this email. " As The Site was Hosted through the Cloudflare hosting provider said no log in our firewall as the site was hosted on Cloudflare. please allowlist Googlebot’s ips so the site can be indexed properly again, if these has been blocked by Cloudflare firewall.


  1. Robots.txt is not blocking anything
  2. Sitemap.xml is not able to refetch
  3. some of the urls are indexed but not in proper format some of meta description is missing some of meta titles are not proper sample of [Cloudflare is not allowing to add Url]


Visit Google Search Console, use the “Inspect URL” tool, then click on the “Test Live URL” button.

After Google finishes fetching the page, if the result is that it was blocked by a 403 head to Cloudflare Dashboard > Security > Events.

Find the event related to this URL block. It will give you details on what Cloudflare service has actually blocked/challenged the request, then you can adjust your settings accordingly, by either editing a WAF Custom Rule to avoid blocking Known Bots, or disabling/skipping services such as (Super) Bot Fight Mode.

If you cannot find a related Security Event, Googlebot is likely being blocked by the origin server. Check your server firewall logs.

This is expected as Google has stopped following meta tags about 3 years ago, please search online.

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.