Cloudflare - caching with Auth

We have a test server that has HTTP authentication.

Cloudflare is being used to do CDN caching.

The test server has got authentication to avoid it being indexed by Google official site (as the robots.txt file occasionally got deleted!).

Will Cloudflare be able to cache content if we have authentication on this server?

If so, how is it doing it? I thought the content would be transmitted directly between the server and the browser.

If the only intention is to avoid it being indexed by Google, you can alternatively add a header or an HTML tag with the “noindex” directive. According to Google documentation, that’s the preferred method, while a robots.txt, even if not deleted, may eventually be bypassed. For instance, if another page links to a URL that’s blocked by robots.txt, it may be indexed, though its content won’t show in the results page.

In an Apache server, you can add a header via .htaccess:
Header set X-Robots-Tag "noindex"

An alternative would be to create a Cloudflare Firewall Rule to block Googlebot altogether, and remove it when the site goes live.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.