Googlebot is unable to access my website sooperchef.pk

Googlebot is unable to access my website sooperchef.pk. can confirm that if any security rule is blocking google bot to access my website . all the pages of my website got de-indexed as google bots are unable to access my website. I have contacted to google support due to deindexing and they told me your security rules are blocking google bot from crawling your website contact to your hosting provider or any other you are using. please let me know why this is happening with the website.

May I ask do you se any blocked request in the Firewall tab → Firewall Events of Cloudflare dashboard for your domain name?

Are you using some custoim Firewall Rules, like country blocking, etc. at Cloudflare?

Furthermore, if you have robots.txt file, have you checked if Google can access and read your sitemap file?

Nevertheless, regarding re-indexing, have you recently switched from HTTP to HTTPS, or non-www to www, or vice-versa?

Did the HTML meta tag robots changed it’s value maybe?

1 Like

my domain name is sooperchef.pk .
i am using custom firewall rule to block specific country and my robots.txt and sitemap.xml files are fine and are readable to the google and i am redirecting my website from http to https and non www too www forcefully and, i am using this mechanism from last year but issue come in this month

Hopefully not the US and/or Google somehow?
Can you share the screenshot of your Firewall Rules?

Do you see any blocked requests under the Firewall → Firewall Events tab at Cloudflare dashboard?

This is not true and not working so far as I see and tested (using below Google Tools).
Kindly, review below as far as I am afraid there is one issue with it:

I would rather atler your robots.txt file to have content like below and give 24-48 hours for Google to read and apply it:

User-agent: *
Sitemap: https://www.sooperchef.pk/sitemap.xml
Disallow: /NewApisFold/
Disallow: /NewmySooopErChefTesting/
Disallow: /home-abc/
Disallow: /public/

Kindly, review this too as far as there are some issues/errors in the redirect configuration:

  • http://www.sooperchef.pk -> https://www.www.sooperchef.pk/ results in an error: NOT OK, TIMEOUT ERROR

I see the <link> canonical is used - which is recommended and okay.

Hm.

As far from the screenshot of your first post, I am worried if Bot Fight Mode is somehow blocking Googlebot or maybe your custom Firewall rule?

I see meta robots HTML tag is implemented too (index, follow), which is ok, but might need to update them to be like:

<meta name="robots" content="max-snippet:-1, max-image-preview:large, max-video-preview:-1">

Viewport is ok, good but I prefer using:

<meta name="viewport" content="initial-scale=1.0, maximum-scale=2.0, width=device-width, user-scalable=yes, minimal-ui">

Interesting, I tried testing and both Google tools from below links report “error” while fetching your Website:

Do you see any blocked request now from Googlebot or Google ASN under the Firewall → Firewall Events tab at Cloudflare dashboard?

Nevertheless, regarding SEO, you might be missing Structured Data markup for your content too:

You can temporary switch from :orange: to :grey: (DNS-only), or enable the “Pause Cloudflare for this site” option from the Overview tab on Cloudflare dashboard.
Therefore, wait for a few minutes.
Test again using above Google Tools and see the results.

Nevertheless, you can allow “good bots” by using the cf.client_bot field in Firewall Rules - in case there are some blocked requests in Firewall → Firewall Events tab at Cloudflare dashboard for your domain name:

Or by creating a custom Firewall Rule and allow Googlebot IP ranges as follows at the link below:

hi here is the screen of my event tab of firewall rules

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.