X-robots-tag not used by googlebot

General Rules
1

Hi all!
I have a website on cloudflare pro plan.
I set a x-robots-tag for pdf files in my htaccess file and with a cloudflare transform rules.
When i dns-only my website, everthings is fine and pdf are not indexed. When i do an url inspect on google search console, pdf files are not indexed.
When i proxified with cloudflare, pdf files are marqued as indexed in google inspector.
I purge the pdf file and it’s the same issue.
Could you help me?

4

That header generally works on Cloudflare. Can you post a sample URL?

1 Like
5

https://www.calendriergratuit.fr/calendrier-pdf/annuel/calendrier-2036-P.pdf

6

The header is sent.

$ curl -I https://www.calendriergratuit.fr/calendrier-pdf/annuel/calendrier-2036-P.pdf | grep robots
x-robots-tag: noindex
7

yes but i’m not sure google can see this header.
When i test it on google inspector,
‘HTTP/1.1 200 OK
accept-ranges: bytes
cache-control: max-age=31536000
content-encoding: mi-sha256-03
content-length: 390653
content-type: application/pdf
digest: mi-sha256-03=P0fb1fDVcjrLSft1GA0ctGoTh27BL6UTuKnbed4Jqpw=
server: cloudflare
vary: Accept-Encoding
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN’

x-robots-tag: noindex is not appearing

If i disable cloudflare with dns-only, it’s ok…

8

I am not sure what “Google Inspector” is, but that would be a question for Google I am afraid. The header itself is sent, respectively forwarded, by Cloudflare.

1 Like
9

There’s also no cache status header, so it seems as if Google did not show all headers.

1 Like
10

thanks!

1 Like
11

There is a discrepancy with the Content-Length header

curl -I https://www.calendriergratuit.fr/calendrier-pdf/annuel/calendrier-2036-P.pdf
HTTP/2 200
date: Thu, 27 Jul 2023 10:32:10 GMT
content-type: application/pdf
content-length: 390309     # <-- Not the same as shown by Google.
# and the rest of the headers
1 Like
12

Hi all,
I add this to my htaccess and it’s working now for pdf

Header set Cache-Control "max-age=0,must-revalidate"

pdf files are now marked as noindex in google search console.

13

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.