Robots.txt file?

I use cloudflare pages. And I don’t have this file in my website folder. Does Cloudflare automatically put this file in the server? If yes, where can I see it and edit it? Thanks


If you want a robots.txt as part of the project you’ll need to add it yourself before deploying/publishing.


Since you’re not having this file, meaning Search engines wouldn’t be able to read it and follow the parameters related to the indexing, etc.

You could leverage Cloudflare products to make sure if there is a request to a /robots.txt, it responds as a text/plain MIME file type with the content you’d want to, that way it would mimic the “real” and a “physical” one like it’s being hosted, while in real life it’s not the case :slight_smile:

1 Like

Thanks. I think what was happening is that whenever Google crawled it was redirected to - and that flagged a duplicate content on my ahref account. So now I will just create manually the robots.txt file

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.