Google robots.txt blocked

[gbro648]

Have a store on Shift4Shop, - .chessset.com, our robots.txt is fine, however, they use cloudflare, and Cloudflare is blocking google bots (google ads, google search google image, etc), error from cloudflare per shift4shop tech is

Cloudflare is reading the request from this specific IP as being a Fake Google Bot 34.83.190.242 (Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/86.0.0.0 Safari/537.36)

How am I, the store SYS Op supposed to get cloudflare to not block google bots?

[paul32]

@gbro648

34.83.190.242

That’s not in Google List of IPs for GoogleBot: https://developers.google.com/static/search/apis/ipranges/googlebot.json

It is GoogleUserContent so probably quite rightly flagged as a fake GoogleBot

[gbro648]

agreed, don’t know where shift4 is getting it from,
the google search console shows on the live test:
Googlebot blocked by robots.txt

Googlebot blocked by robots.txt


Googlebot blocked by robots.txt
Script
https://googleads.g.doubleclick.net/pagead/viewthroughconversion/1070036369/?random=1692397912107&cv=11&fst=1692397912107&bg=ffffff&guid=ON&async=1&gtm=45be38g0&u_w=412&u_h=732&url=https%3A%2F%2Fwww.chessset.com%2F&hn=www.googleadservices.com&frm=0&tiba=Rochester%20Chess%20Center%20-%20Chess%20Sets%2C%20Boards%2C%20Clocks&auid=523418798.1692397912&data=event%3Dgtag.config&rfmt=3&fmt=4

so again, google’s claiming their bot is blocked, but my robots.txt shows

2# Disallow all crawlers access to certain pages.
3
4 User-agent: *
5Disallow: /checkout.asp
6Disallow: /add_cart.asp
7Disallow: /view_cart.asp
8Disallow: /error.asp
9Disallow: /shipquote.asp
10Disallow: /rssfeed.asp
11Disallow: /admin/
12Disallow: /apple-app-site-association
13
14User-agent: Googlebot
15Disallow:
16
17User-agent: AdsBot-Google
18Disallow:
19
20User-agent: Googlebot-Image
21Disallow:
22
23User-agent: Storebot-Google
24Disallow:
25
26User-agent: Google-InspectionTool
27Disallow:
28
0 Errors 0 Warnings

So the hunt continues,
Thanks for the reply !