Search Console "Sitemap couldn't fetch error"

Hi Sir / Madam, Hope you are doing good.

I’m facing some errors on my search console. When I’m trying to submit a sitemap on the search console it does not submit successfully and shows an error as “Sitemap couldn’t fetch” when I open the error it says General HTTP error.

When I went deeper into this issue I found the website from the backend showing a “503 Service Unavailable” error.

So why is it showing so? The website is opening properly but why server from the backend showing a 503 error, because of that I can’t submit a sitemap, which causes no indexing of any blog URL.

When I describe the same issue with the Search Console support system they said this

"
The sitemap is fine but Googlebot can’t reach the site

When I check with a spoofing tool it shows that a status 503 is returned to user agent Googlebot

On the other way, the mobile-friendly tool is showing (crawl failed’ without details so Googlebot can’t reach it at all or don’t get an answer from your site

Like I advised before, check the server logs to see if you see traffic from Googlebot and what status it returns, check if it also returns a 503

If you don’t have access to the server logs, then check with your web host

Also check Cloudflare firewalls, security plugins, .htaccessn, or other server settings."

This was their reply, Now what to do?

Greetings,

Thank you for asking.

Did you checked for any relevant request being shown as “challenged” or “blocked” at Firewall tab - Overview?

Reply here with feedback information:

  1. Security Level - low, med, high, i am under an attack, which one is set?
  2. Bot Fight Mode - enabled or disabled
  3. Firewall Rules - do you use some or none
  4. Is the sitemap physically existing or has to be generated by the server
  5. robots.txt file - what does it contain
  6. Page Rules - do you use some or none
  7. Have you checked if your sitemap XML file is valid?
  8. Is it being served over HTTPS?

If so, I’d suggest temporary Pausing Cloudflare for better troubleshooting at your origin host/server.

  • Use the “Pause Cloudflare on Site” option from the Overview tab for your domain at dash.cloudflare.com .
  • The link is in the lower right corner of that page.
  • Give it five minutes to take effect

It might be if the sitemap file needs to be generated, and if the requests takes long time.

Hi, Thank you so much for replying.

I’m answering all your questions below.

Did you check for any relevant request being shown as “challenged” or “blocked” at the Firewall tab - Overview?
Ans: Yes it shows JS Challenge, and also the country mentioned with the IP address.

1. Security Level - low, mid, high, I am under an attack, which one is set?
Ans: Essentially Off

2. Bot Fight Mode - enabled or disabled?
Ans: Enabled

3. Firewall Rules - do you use some or none?
Ans: I have created 2 firewall rules 2 months before, cause at that time I was getting a lot of spam traffic on my website, but yesterday I’ve deleted both firewall rules, I thought this might be stopping google bots to crawl my site so I’ve deleted it yesterday. The rules I created, I do not remember properly but it’s something like Tor.

4. Is the sitemap physically existing or has to be generated by the server?
Ans: I didn’t get this question properly, but if you’re asking whether the sitemap has been generated or not? Then yes, it has been generated and exists properly, and I’ve made it from Rank Math SEO Plugin, here is the sitemap link: https://growwpedia.com/sitemap_index.xml/

5. robots.txt file - what does it contain?
Ans:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://growwpedia.com/sitemap_index.xml

6. Page Rules - do you use some or none?
Ans: None

7. Have you checked if your sitemap XML file is valid?
Ans: I’ve used this website XML sitemap validator to check if my sitemap XML file is valid or not and It shows “No issue detected”. See the Screenshot below.

8. Is it being served over HTTPS?
Ans: Yes, my website URL starts from HTTPS

If so, I’d suggest temporary Pausing Cloudflare for better troubleshooting at your origin host/server.

Ok, I will try to do this as well and check if it works.

Meanwhile check the above answers I’ve given and reply back if you’ve any more solution to solve this problem, Your help will be appreciable.

Waiting for your positive reply.

Regards

Hi,

I’ve tried this that you said " If so, I’d suggest temporary Pausing Cloudflare for better troubleshooting at your origin host/server."

Now the sitemap is submitted successfully without any error, See below Screenshot.

But when I clicked on this, then 2 URLs are still showing the " sitemap Couldn’t fetch " error for Pages and Posts. Check the below screenshot for a better understanding of what I’m saying.

Then when I inspect the live URL test for /post-sitemap.xml one error is showing see the below screenshot.

Now how to get rid of this issue " No: ‘noindex’ detected in ‘X-Robots-Tag’ http header "

Waiting for your reply.

Are you using WordPress?
Please check WordPress Settings > Reading and uncheck the “Search engine visibility”.

Nevertheless, to Google, if you are using Yoast SEO, be sure to submit only sitemap_index.xml → from the screenshot, you have submitted it with the trailingslash at the end / like /sitemap_index.xml/

Furthermore, it’s normal for “inner sitemaps” to be “noindex”.

It might say “error at first”, leave it for a day, then re-check because it knows to be some issue at least was for me.

Check your robots.txt file, post content of it here.
Check your htaccess file, maybe there is the header which you should remove.

Otherwise, using Transform Rules at Cloudflare dashboard you can remove that HTTP header.

Hi, Thanks for replying

Please check WordPress Settings > Reading and uncheck the “Search engine visibility”.

It is already unchecked.

if you are using Yoast SEO, be sure to submit only sitemap_index.xml → from the screenshot, you have submitted it with the trailing slash at the end / like /sitemap_index.xml/

I’m using the rank math SEO plugin. And Yes, I’ve made changes according to what you said.

Furthermore, it’s normal for “inner sitemaps” to be “noindex”

Ook, but why my Posts were not showing on Google? It hasn’t been indexed again, and in the discovered URL also it is showing 0.

Check your robots.txt file, post the content of it here.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://growwpedia.com/sitemap_index.xml

Now, My Upper sitemap has been successfully submitted and inside it, the sitemap for the page URL also got submitted but only the sitemap of the post URL hasn’t been submitted yet, as shown in the below screenshot.

When I inspected the live test of this post sitemap URL, that is of this /post-sitemap.xml
it again shows this error.

Now what to do to get solve this error?

Waiting for your reply.

Might have to wait for few hours until Google updates that information.

First fetch can be always error as XML file has to be generated “on-the-fly” because of the plugin, it does not physically exist yet.

Ok, I will wait till tomorrow or maybe till 2 days and after then will let you know what is the status.

Meanwhile, Thank you for your support and guidance, you are truly talented and expert. Keep it up.

1 Like

HI, Fritex

It’s been more than 24 hrs gone, But in the search console, it still shows the same error. See the screenshot below.

What to do? Waiting for your reply

Thanks

Please reply,

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.