I’m using a Pro subscription. When Google tries to crawl my website it gets “noindex” in header of the server answer by Cloudflare. Due that my site is not listed by Google.
How I can disable this behaviour of Cloudflare?
Just to be sure - have you verified that your origin server does not send this header? i.e. does it happen when you either make the request directly to your server, or, disable CF for the host by clicking the orange cloud in the DNS tab, temporarily for the test?
Tested with disabled Cloudflare proxy and curl sending Googlebot user agent, this is the header I get:
> GET / HTTP/1.1
> User-Agent: Googlebot/2.1 (+http://www.google.com/bot.html)
> Accept: */*
> Host: *censored*
>
< HTTP/1.1 200 OK
< Date: Fri, 21 Dec 2018 10:48:43 GMT
* Server Apache is not blacklisted
< Server: Apache
< Strict-Transport-Security: max-age=15552000
< Last-Modified: Wed, 26 Sep 2018 06:04:05 GMT
< ETag: "2a69-576bffd65efbb"
< Accept-Ranges: bytes
< Content-Length: 10857
< Vary: Accept-Encoding
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< Content-Type: text/html
noindex is not beeing sent. Strangely also no noindex when browsing it with enabled CF and browser.
Just Google complains about noindex when crawling the page (crawling errors in search console).
Due of that, the website won’t appear in Google index.
My robots.txt content:
User-agent: *
Disallow:What is your domain to begin with?
Where did you get that header?
I get that when doing curl on the webserver itself:
curl -v -H "Host: capsop.com" --insecure --user-agent "Googlebot/2.1 (+http://www.google.com/bot.html)" https://127.0.0.1Doesnt that sort of prove that Cloudflare is NOT involved?
Do you see somewhere noindex when no CF is beeing used?
Google just alerts me when it tries to crawl the site.
.htaccess:
ErrorDocument 404 /404.html
ErrorDocument 403 /403.html
RewriteEngine on
RedirectMatch 301 /lightaidra-cc-investigation$ /itsec/2014/05/17/lightaidra-cc-investigation.html$1
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_URI} /$ [NC]
RewriteCond %{REQUEST_FILENAME} .*\.(png|jpg|gif|jpeg|html|htm)$ [NC]
RewriteRule ^(.*)/$ https://%{HTTP_HOST}/$1 [R=301,L]
in VirtualHost config:
Header set X-Robots-Tag “none”
So where my server sends noindex which is beeing forwarded to CF?
Considering I didnt get such a header when going via Cloudflare I wouldnt think there is one directly either.
Can you post the actual official URL where you get that header?
Oh, my fault. I think I found the error:
Header set X-Robots-Tag all
Was set in additional config which I cross loaded to Apache.
I requested a new crawl sequence, maybe it works now. (Will now in some days as Googlebot takes it time)
This topic was automatically closed after 14 days. New replies are no longer allowed.