Cloudflare Managed Challenge & Google Indexing

[wedesign]

What is the name of the domain?

wedesign.lu

What is the error number?

403

What is the issue you’re encountering

Pages are not indexed with Cloudflare activated

What steps have you taken to resolve the issue?

Pause Cloudflare

What are the steps to reproduce the issue?

I created a custom rule called “Country Block” to give all visitors outside of Luxembourg a managed challenge. I noticed that my page can not be found in google search anymore and saw in Google Search Console that no pages are indexed. Uploading a sitemap returned a 403 error. I paused Cloudflare and managed to upload the Sitemap with no problems.

What I read from similar topics here in the forum is that people say the problem doesn’t come from Cloudflare itself, so I am wondering which settings I have to adjust to not block “good bots” and still prevent unusual (scam) traffic from countries, thus giving a managed challange to all but luxembourgish visitors.

[fritex]

@wedesign

I created a custom rule called “Country Block” to give all visitors outside of Luxembourg a managed challenge. I noticed that my page can not be found in google search anymore and saw in Google Search Console that no pages are indexed. Uploading a sitemap returned a 403 error.

This is an expected behaviour and event for the Custom WAF rule which you’ve applied with expected error code.

@wedesign

so I am wondering which settings I have to adjust to not block “good bots” and still prevent unusual (scam) traffic from countries, thus giving a managed challange to all but luxembourgish visitors.

If you’d want only Google and Bing to index your submitted sitemap.xml file to the Google Search Console/Bing Webmaster Tools and visit robots.txt file, while block everyone else (including yourself) to access the same file, I’d suggest looking at by post from below:

FW rule blocking Microsoft ASN without blocking Bing

You might want to switch the toggle button from green to gray. Therefore, make sure the method is “block”. However, I might not be sure how Cloudflare checks this, by user-agent or IP range So, maybe like writing your own Firewall Rule … for example: (http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 15169 and not http.user_agent contains "Googlebot") or (http.request.uri.path contains "sitemap" and ip.geoip.asnum ne 8075 and not http.user_agent contains "Bingbot") or…

So the 1st rule would be as from above post, while the 2nd rule would me Managed Challenge action for everyone except if country is not Luxemburg.