Does cloudflare cause issue with google merchant feed checks?

We have been experiencing an intermittent increase in disapproved google merchant center products. 56% and steadily climbing.
the merchant center errors are :
Unavailable mobile landing page & Unavailable desktop landing page
The error detail is mostly: “Hostname not resolvable” however there are a few “Timeout on DNS lookup”

It seems the google bots checking the validity of feed details are blocked, is this possible with cloudflare settings ?

We have restricted access (with captcha) by country to US, UK, IE
Also limited the access by 50pages in 1 min.

thanks

I have exactly the same issue

Getting lots of hostname not resolvable messages in Google Merchant Centre over the last 2 weeks

So you even challenge Googlebot and possibly block crawling and indexing your Website? :thinking:

Ouch! Well then, that’s the expected result. :roll_eyes:

You could try whitelisting and allowing Google by it’s user-agent sring, or either by the IP range list, or at least a temporary solution for testing purpose to try with the whole Google AS number (AS15169) - but, if you do it for permantent, it could be a bad thing as far as there are also “bad bots” and “crawlers” you really do not want to scrape your Website.

  • using IP Access Rules and Firewall Rules in combination …

See below:

Or it could be to the Bot Fight Mode, if so.

Nevertheless, isn’t it caused by your origin host / server robots.txt file, if so?:

Could you explain how to do this in more detail?

We have no restricted access like the user66910 but are having the same issues

Thanks

Sure, step-by-step in case if need :wink:

You can use above links (for IP ranges) or the AS number AS15169 (Google - hopefully they use this, as they have multiples).

For Firewall Rules:

  1. Log in to Cloudflare dashboard using your Cloudflare account
  2. Open up the Firewall tab → select Firewall Rules
  3. Click on the button Create a Firewall Rule
  4. Give it a name, select AS Num equals and enter Google’s AS number 15169 (or rather the IP ranges)
  5. Change the action to “Allow” from the dropdown menu and click the “Deploy” button

The otherway using IP Access Rules would be to (either add only the Google’s AS number AS15169, or IP ranges):

  1. Log in to Cloudflare dashboard using your Cloudflare account
  2. Open up the Firewall tab → select Tools
  3. Under the IP Access Rules, enter them change “Block” to “Allow” and click on the button “Add”

With the image:

You can use below expression (AS number or IP ranges) if using Firewall Rules:

  • (ip.geoip.asnum eq 15169)

  • (ip.src in {2001:4860:4801:10::/64 2001:4860:4801:11::/64 2001:4860:4801:12::/64 2001:4860:4801:13::/64 2001:4860:4801:14::/64 2001:4860:4801:15::/64 2001:4860:4801:16::/64 2001:4860:4801:17::/64 2001:4860:4801:18::/64 2001:4860:4801:19::/64 2001:4860:4801:20::/64 2001:4860:4801:21::/64 2001:4860:4801:22::/64 2001:4860:4801:23::/64 2001:4860:4801:24::/64 2001:4860:4801:25::/64 2001:4860:4801:2::/64 2001:4860:4801:26::/64 2001:4860:4801:27::/64 2001:4860:4801:28::/64 2001:4860:4801:29::/64 2001:4860:4801:2a::/64 2001:4860:4801:2b::/64 2001:4860:4801:2c::/64 2001:4860:4801:2d::/64 2001:4860:4801:2e::/64 2001:4860:4801:2f::/64 2001:4860:4801:30::/64 2001:4860:4801:31::/64 2001:4860:4801:32::/64 2001:4860:4801:33::/64 2001:4860:4801:34::/64 2001:4860:4801:35::/64 2001:4860:4801:3::/64 2001:4860:4801:36::/64 2001:4860:4801:37::/64 2001:4860:4801:38::/64 2001:4860:4801:39::/64 2001:4860:4801:3a::/64 2001:4860:4801:3b::/64 2001:4860:4801:40::/64 2001:4860:4801:41::/64 2001:4860:4801:42::/64 2001:4860:4801:43::/64 2001:4860:4801:44::/64 2001:4860:4801:45::/64 2001:4860:4801:46::/64 2001:4860:4801:47::/64 2001:4860:4801:48::/64 2001:4860:4801:49::/64 2001:4860:4801:4a::/64 2001:4860:4801:50::/64 2001:4860:4801:51::/64 2001:4860:4801:53::/64 2001:4860:4801:60::/64 2001:4860:4801:61::/64 2001:4860:4801:62::/64 2001:4860:4801:63::/64 2001:4860:4801::/64 2001:4860:4801:64::/64 2001:4860:4801:65::/64 2001:4860:4801:66::/64 2001:4860:4801:67::/64 2001:4860:4801:68::/64 2001:4860:4801:69::/64 2001:4860:4801:6a::/64 2001:4860:4801:6b::/64 2001:4860:4801:6c::/64 2001:4860:4801:6d::/64 2001:4860:4801:6e::/64 2001:4860:4801:6f::/64 2001:4860:4801:70::/64 2001:4860:4801:71::/64 2001:4860:4801:72::/64 2001:4860:4801:73::/64 2001:4860:4801:74::/64 2001:4860:4801:75::/64 2001:4860:4801:76::/64 2001:4860:4801:77::/64 2001:4860:4801:80::/64 2001:4860:4801:81::/64 2001:4860:4801:82::/64 2001:4860:4801:83::/64 2001:4860:4801:84::/64 2001:4860:4801:85::/64 2001:4860:4801:86::/64 2001:4860:4801:90::/64 2001:4860:4801:91::/64 2001:4860:4801:c::/64 2001:4860:4801:f::/64 66.249.64.0/27 66.249.64.128/27 66.249.64.160/27 66.249.64.192/27 66.249.64.224/27 66.249.64.32/27 66.249.64.64/27 66.249.64.96/27 66.249.65.0/27 66.249.65.128/27 66.249.65.160/27 66.249.65.192/27 66.249.65.224/27 66.249.65.32/27 66.249.65.64/27 66.249.65.96/27 66.249.66.0/27 66.249.66.128/27 66.249.66.192/27 66.249.66.32/27 66.249.66.64/27 66.249.68.0/27 66.249.68.32/27 66.249.68.64/27 66.249.69.0/27 66.249.69.128/27 66.249.69.160/27 66.249.69.192/27 66.249.69.224/27 66.249.69.32/27 66.249.69.64/27 66.249.69.96/27 66.249.70.0/27 66.249.70.128/27 66.249.70.32/27 66.249.70.64/27 66.249.70.96/27 66.249.71.0/27 66.249.71.128/27 66.249.71.160/27 66.249.71.192/27 66.249.71.32/27 66.249.71.64/27 66.249.71.96/27 66.249.72.0/27 66.249.72.128/27 66.249.72.160/27 66.249.72.192/27 66.249.72.224/27 66.249.72.32/27 66.249.72.64/27 66.249.72.96/27 66.249.73.0/27 66.249.73.128/27 66.249.73.160/27 66.249.73.192/27 66.249.73.224/27 66.249.73.32/27 66.249.73.64/27 66.249.73.96/27 66.249.74.0/27 66.249.74.32/27 66.249.74.64/27 66.249.75.0/27 66.249.75.128/27 66.249.75.160/27 66.249.75.192/27 66.249.75.224/27 66.249.75.32/27 66.249.75.64/27 66.249.75.96/27 66.249.76.0/27 66.249.76.128/27 66.249.76.160/27 66.249.76.192/27 66.249.76.224/27 66.249.76.32/27 66.249.76.64/27 66.249.76.96/27 66.249.77.0/27 66.249.77.32/27 66.249.77.64/27 66.249.79.0/27 66.249.79.128/27 66.249.79.160/27 66.249.79.192/27 66.249.79.224/27 66.249.79.32/27 66.249.79.64/27 66.249.79.96/27})

With the image (IP Access Rules) - possibly you cannot enter the ranges due to the limitation, and would have to “transform” them into even more /24 by CIDR notation:

Another way to try out would be to select cf.client.bot in Firewall Rules and set to Allow as far as “good bots” are recognized (including Google) as follows on the below link:

Or some kind of combination of it.

Thank you so much for the detailed reply!

I can see events like the below in my Firewall. Assume this would be the Google Merchant Crawler:

Should this have a path to: /wp-admin/admin-ajax.php ?

From the above screenshot I see Service: Bot fight mode.

I just wonder, what happens if you temporary disable it for testing purpose?:

Have turned this off and also added the IP rule and cf.client.bot rule you posted above

Fingers crossed that will solve it

May I ask, is the URL from above screenshot (the Query string part) which Google Merchant bot/crawler visits always the same or it changes?

Furthermore, from my experience it seems to me that a request came from a Google server? and also with the user-agent of WordPress itself - like it was run by wp-cron - I mean, seems to me as the request from which came from, the WordPress is hosted on that Google server too? - Could be I am wrong about it.

I can see different query strings. Another example being: ?doing_wp_cron=1636578464.0099220275878906250000

I dont know if Firewall events are the Google Merchant Bot> I am just assuming they are since they say they are from
AS15169 GOOGLE

Hi all, for anyone reading this I got the below from Siteground Support today, seems like an issue their end:

We have detected a connectivity issue affecting the Google platform being able to connect to websites using the SIteGround DNS service. this was detected yesterday and currently our team is working on a solution.

They have confirmed the issue is effecting Google Mobile Checker and Google Ads

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.