Robots.txt - how to disable CF on this file + Rendertron

I have robots.txt file with
User-Agent: *
Allow: /
Disallow: /pl$

How do I disable Cloudflare in this file ? I got rendertron (dynamic rendering pages) - so I have to disable rendertron on that file also

I added do page rules :
domain_com/robots.txt
Disable Security
but I got still this problem.

I saw similar topics but that cases hasn’t got rendertron…

Cloudflare add code and rendertron gives it to googlebot:

<html><head><base href="domain_com/"></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">
User-Agent: *
Allow: /
Disallow: /pl$
</pre><script defer src="static.cloudflareinsights_com/beacon.min.js/vaafb6**" integrity="sha512-0ahDY***==" data-cf-beacon='{"rayId":"76**","token":"de2a2d5***","version":"2022.11.0","si":100}' crossorigin="anonymous"></script>
</body></html>

Are you sure that’s robots.txt? It looks more like it’d be robots.html

Robots.txt should be a plain text file with no HTML markup. Remove all the tags, and you should be fine.

I know what should be in robots.txt and I wrote that I add:
User-Agent: *
Allow: /
Disallow: /pl$

but rendertron is cacheing files and CF add script
(html tags was added by rendertron - and I have to find also how to disable it…)

I can remove tags and CF bec rendertron will rewrite it for example next day…

I was going to suggest that your question may be more effective if directed toward rendertron, but the developer appears to have deprecated it.

A better solution would be to create a Worker that contains your Robots.txt, then add it as a Route for example.com/robots.txt

1 Like

…I know I know…
But I’m asking how disable CF on files like robots.txt