Cloudflare robots.txt Service

[pete.batin]

What is the name of the domain?

notapplicablecom

What is the issue you’re encountering

No documentation for robots.txt service

What is the current SSL/TLS setting?

Full (strict)

What are the steps to reproduce the issue?

Firstly, this form is irrelevant to most issues.

Cloudflare has a feature “Manage bot traffic with robots.txt” which seems like it could have some benefit. However, it replaces the existing robots.txt file. So how do you go about customising it? Is it possible to customise it? Why would anyone want to replace their existing robots.txt file with Cloudflares if it cannot be edited?

There are pages that I don’t want search engines to index, I want to be able to specify the locations of sitemaps and control crawl rates. I can’t find any documentation for this service.

[cscharff]

@pete.batin

Why would anyone want to replace their existing robots.txt file with Cloudflares if it cannot be edited?

You wouldn’t.

The vast majority of Cloudflare customers have little to no technical knowledge. Like the “I’m under attack button”, the robots.txt feature is an ‘easy button’ to put a file in place to prevent AI bots from visiting a site and not useful for more complex needs.

It’s primarily a marketing feature… made for a flashy blog post and some press about how Cloudflare helps the little guy stand up to big evil AI corporations.

For technical users or users with more need than the simple file created, creating and managing the features by hand in your own txt file is the appropriate path.