Google console is not able to fetch sitemap. Wordfence + Cloudflare

General
1

Hello Cloudflare,

My apologies in advance if this topic is in the wrong section. Please move if needed.

As the tittle states, google console is not able to fetch my sitemap azpaintingvancouver.ca
I am using wordfence plugin with cloudflare.

Currently this is what i have done;
robot.txt file is set and a link to sitemap is provided in the txt file.
sitemap azpaintingvancouver.ca/sitemap_index.xml is recognizable
I have set google verification code TXT record in DNS of my cloudflare
I have connected my site to wordfence and have set the IP’s in my in cloudflare firewall rules.
In wordfence setting I have enabled CF-Connecting-IP and I have added the ip address to my DNS firewall rule of cloudflare. Not sure if i had to add the CF connection ip to firewall, but i did just to see if google console would start fetching.

I am out of ideas what can cause the robots not to fetch the site and would appreciate any lights shinned on this.

Thank you in advance!!!

2

Few things to consider:

I also run couple of WordPress Websites with Wordfence and behind Cloudflare, while having Yoast SEO (sitemap_index.xml file auto-generated) - all working fine.

May I ask have you got any Firewall Rule or Security Options at Cloudflare dashboard being enabled which could potentially block the Googlebot from reading your robots file and accessing your sitemap file?

I am afraid your robots.txt file is not correctly structured and/or written well.

The robots.txt file from:
https://azpaintingvancouver.ca/robots.txt

Contains:

User-agent: Googlebot
Disallow:

sitemap: https://azpaintingvancouver.ca/sitemap_index.xml

But, from my point of view, it should be like:

User-agent: *
Sitemap: https://azpaintingvancouver.ca/sitemap_index.xml

Disallow: /wp-admin/
Disallow: /cgi-bin/
Disallow: /trackback/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Allow: /wp-admin/admin-ajax.php

Regarding this:

User-agent: Googlebot
Disallow:

If you want to disallow some bot to index your sitemap, it should be written like:

User-agent: 360Spider
Disallow: /
4 Likes
3

Thank you for your suggustions @fritex

i do have four seperate firewall rules set. Three of them is for wordfence ip’s
and i have also added the CF-Connecting-IP address to firewall rules.

All rules are set to allow.

Yoast did not create a sitemap automatically. I downloaded my sitemap from another website which creates sitemaps of your website.

The robots.txt file was also not in my list and I had to create that and upload it.
little bit of google search and i found that the codes in there would allow the robots to index the sitemap and everything esle in my folders.

I guess i will update my robot.txt to what you have written.

Also, thank you for helping a total noob here

4

This morning i noticed that my sitemap is not being fetched. Thank you in advance!

5

Hello,

I was hoping to find out if I would need to set the cloudflare ip’s in my allowlist in wordfence in order for my sitemap to be accusable?

Thank you in advance!

6

Not likely. I use Wordfence and haven’t had problems with it blocking sitemaps. And if it was blocking anything, it’d show up in the Live Traffic page. Not only that, but you most likely have Wordfence correctly configured to figure out your visitor’s IP addresses, so Cloudflare IP addresses won’t matter.

1 Like
7

Thank you for your replay @sdayman
I am latterly out of ideas what it can cause the sitemap not to be fetched by google search console.
Thank you for verifying my questions.

I can see the live traffic but i can’t find any blocks for google bots.
url inspection in google search console does show that the sitemap is valid

url inspection
url inspection1426Ă—835 48.6 KB

8

@sdayman i am not sure if i should drop a picture of the live traffic here. but i do see ip address and or this inside ip address column IP: 2406:da18:846:3500:5761:f388:5eaa:2a8f

9

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.