Cloudflare is not friendly with google shopping crawl

When use CDN, google merchant center can’t crawl all links, it shows http 4xx error.

1, Use Cloudflare CDN, google merchant can’t crawl url and get http 4XX error.
2, Turn of cloudflare CDN, and only use DNS, everything became to well, no error.
3, I have ever used Akamai CDN, never met any error, including this error.
4, Google service can’t give out the exact error code, i don’t know it is 403 or 404.

I submitted a ticket 1837392, but nobody want to solve this matter, only thought the matter is caused by my server, that’s a joke. And reply any information after 24 hours, i don’t know if cloudflare can solve this matter within one year according to the customer service efficiency, so i have to ask for help through community.

Hi @lihuanwu,

This issue was raised a few days ago and is already being addressed by the Cloudflare team.

Cc: @cloonan @mdemoura

Is there any method to don’t block ip address by cloudflare default? I turned off WAF and firewall rules, this error was not solved, and i can’t find where can set this, so i think some ip addresses are blocked by cloudflare system default.

The customer service reply any information after 24 hours, it is too slow, and i found some other users also met the same situation, but all can’t give out method through google.

If you can provide more details about the requests that are being blocked, we could try to draft a temporary firewall rule in order to accept them while a definitive/official solution isn’t deployed.

What are the source IPs and the User Agent strings?

1 Like

I contacted google but got no reply about their ip addresses.


Unfortunately, the data available in the screenshots do not inform any of the necessary information.

Could you please check the firewall events in the Cloudflare Dashboard?

1 Like

No firewall events, so i think it is caused by cloudflare system, not WAF.

In that case it’s difficult to draw up a firewall rule in an assertive manner.

Do the blocked requests concern data feeds, products, images, or all of them?

Have you made sure the problem is not being caused by your robots.txt file?

I have said, if only use DNS, everything is OK, no error.

And i used Akamai CDN before cloudflare CDN, everything is also OK, no error. Because many ip addresses are blocked by China government, difficult for us to visit website, i have to change to try another CDN.

Only one possibility, there is something wrong setting from cloudflare, but i don’t know how to set it or don’t have the button to set it, because after turn off WAF and firewall rule, the error still exists.

Hi @lihuanwu, I see the was escalated to our Engineering Team and they are looking into it. I’ll add myself to the ticket to track progress.

1 Like

This problem has not yet been resolved, and no one has yet come up with a solution. They all think this error is caused by my server, but nobody explain the 3 questions:

1, only used DNS, everything is OK, no error.
2, I have ever used Akamai CDN, also no error, everything is OK.
3, All visitors can view normally, easy to view all pages, only google can’t visit my website.

What is the current status of your support ticket?

The status is OPEN

You could reply to this post if you can’t / don’t want to wait for a support response.

It’s been over ten days and it hasn’t been resolved

Anybody can help me?

The same issue consist with me also,
I’ve started use cloudflare for 3 days and google merchant center began to mark and disallow with 4xx Error on my products,
Url’s : correct,
Sites loading correct,
Visitors see : correct
But google merchant gives 4xx some how,
Any idea or any body to solve issue ?
Regards

@cloonan @mdemoura

1 Like

The issue at hand is different than Please add a firewall rule for Google Shopping, given that @lihuanwu reports having no Firewall Events.

Can you check your server logs for requests from Google’s IPs?

@msvml have you opened a support ticket?

2 Likes