I use cloudflare pages. And I don’t have this file in my website folder. Does Cloudflare automatically put this file in the server? If yes, where can I see it and edit it? Thanks
No.
If you want a robots.txt as part of the project you’ll need to add it yourself before deploying/publishing.
2 Likes
Since you’re not having this file, meaning Search engines wouldn’t be able to read it and follow the parameters related to the indexing, etc.
You could leverage Cloudflare products to make sure if there is a request to a /robots.txt, it responds as a text/plain MIME file type with the content you’d want to, that way it would mimic the “real” and a “physical” one like it’s being hosted, while in real life it’s not the case 
1 Like
Thanks. I think what was happening is that whenever Google crawled websitevanilla.com/robots.txt it was redirected to websitevanilla.com - and that flagged a duplicate content on my ahref account. So now I will just create manually the robots.txt file
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.