How to Disable Cloudflare's Implemented robots.txt File?

Hi,

I just got done talking with my host’s backend developers and they stated the robots.txt is being served by Cloudflare.

How do I disable that function? I want to create my own.

Thanks and regards,

Tug

I haven’t seen and experienced that.
I mean, served - yes if it exists in both physical or virtual way (if I can call it like that).

But not in a case where Cloudflare would impose its own robots.txt file for your domain generated under the unknown circumstances as far as Cloudflare does not host files.

Maybe you are using some of Cloudflare Apps to generate that?
If so, navigate to Cloudflare Apps at the Cloudflare dashboard and re-check if so.

Otherwise, I am afraid if th file is not being physically present at the origin host / server, therefore it’s being generated and obviously can be accesses via the URL scheme from the application itself (example of Yoast SEO plugin for WordPress, which creates a virtual accessible robots.txt file which the end-user cannot modify - at least, not as the physical file at the server). Usually, the robots.txt file should be located in the root directory of your website/domain.

If using some thirtd-party implementation, maybe that app is creating its own robots.txt file.

2 Likes

Thank you for that quick and detailed response!

I checked out Yoast and they did not have an option for the virtual robots.txt nor did they have the option to edit the file which is weird.

The odd thing was there was no robots.txt file on my server and there was no plugin that was adding it and I am not using any Cloudflare app.

So I created the file and then set it to bypass the cache on Cloudflare and WP Rocket and then all of a sudden it populated what I put in. Weird. It seems like it was a caching issue. A mystery. Somebody get Columbo on the case ASAP.

Thanks again for your insight!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.