I'm having trouble with X-Robots-Tag

[TonyMK]

My WP website uses cloudflare accidentally, for some reason, cloudflare added x-robots-tag: noindex, follow to my sitemap.

As a website, I dropped the index and couldn’t index posts anymore.
I read a document on the internet that said it was due to security, so Cloudflare accidentally added it by itself.

Currently the response of my wp site map is:

HTTP/1.1 200 OK
Date: Mon, 28 Aug 2023 22:30:41 GMT
Content-Type: text/xml; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding,Cookie,User-Agent
X-Content-Type-Options: nosniff
x-robots-tag: noindex, follow
Server: cloudflare
Content-Encoding: br
alt-svc: h3=“:443”; ma=86400

I tried creating a text file _header in public_html
/static/*
Access-Control-Allow-Origin: *
X-Robots-Tag: none

like useless. The site map of my website is still noindexed.

Everyone who has sent this error please point me.

How to completely remove X-Robots-Tag from my website.
Please just me thank you very much.

[cbrandt]

@TonyMK

I read a document on the internet that said it was due to security, so Cloudflare accidentally added it by itself.

That’s totally false. Cloudflare does not manage your site’s x-robots-tag headers.

You need to check your WordPress installation instead, or ask for help at Forums | WordPress.org. Also, I don’t think a sitemap’s x-robots-tag header would influence indexing, but that’s an issue for Google Search Central Community.

[TonyMK]

Thanks for answering.
So can I create any expression rule to disable x-robots-tag header completely using cloudflare?
Please just me thank you very much.

[cbrandt]

You can use Transform Rule - Modify Response Header, as in the example linked below, matching the URI Path for your sitemap’s actual path.

Response header modification examples · Cloudflare Rules docs

[TonyMK]

Thank you very much for your reply.
I’ve tried creating as a documentation template
And I get this error message.

remove’ is not a valid value for operation because it cannot be used on header beginning with ‘cf-’ (Code: 20087)

I do not understand.
The instruction form clearly states “Header name: cf-connecting-ip”

The following HTTP response header modification rule removes the cf-connecting-ip header from the response:

Text in Expression Editor:

starts_with(http.request.uri.path, “/private/”)

Selected operation under Modify response header: Remove

Header name: cf-connecting-ip

When I enter Header name: cf-connecting-ip I get the above error message.

What should I do to fix it?

[cbrandt]

@TonyMK

The instruction form clearly states “Header name: cf-connecting-ip”

That is an example, but it’s not the header you want to remove. Don’t you want to remove x-robots-tag? That’s the name you should input in that field.

[TonyMK]

I have added deleted x-robots-tag as shown.

As when checking the sitemap link on google webmaster tools there is still x-robots-tag.

HTTP/1.1 200 OK
Date: Tue, 29 Aug 2023 19:23:49 GMT
Content-Type: text/xml; charset=UTF-8
Vary: Accept-Encoding,Cookie,User-Agent
X-Content-Type-Options: nosniff
x-robots-tag: noindex, follow
Server: cloudflare

What should I do next to remove x-robots-tag completely.

[cbrandt]

URI Path always starts with a “/”:

[TonyMK]

Thank you very much.
I did it.
Like can you tell me more?
uri.path is to remove the x-robots-tag for a permanent link.
So I want to make sure all other links including post , page , cate , tag , search , … have x-robots-tag removed, am I using the hostname filter field?

(starts_with(http.host, “domain.com”))