Robots.txt issue


#1

Hi,
We recently started to use cloudflare with basic plan for now, but we run into problem. robots.txt file served by cloudflare is different from what we have on our origin server.
Can we control it somehow ?
Thanks


#2

Cloudflare doesn’t change your robots.txt file. Is it possible Cloudflare has an older version cached and it simply needs to be purged to reflect the updated version?


#3

Hi,
Thanks for this. We did purge, but google still not picking this up. It maybe down to google indexing robots.
We also found that we can’t verify ownership of site using for example html tag which we put in place (we can see it in source code of cruise118.com) but for some bizarre reason google asking us to confirm ownership of this:
https://dn1i8v75r669j.cloudfront.net
Is it something we doing wrong ?
Thanks - Matt.


#4

This topic was automatically closed after 31 days. New replies are no longer allowed.