Feature suggestion

First of I would like to thank Cloudflare providing such an amazing service.

I would like to suggest a feature suggestion if I may.

I have noticed sometimes search engines and other crawler types don’t obey items such as robots.txt from time to time.

In my case it was a “Noindex” instruction not being obeyed and a SEO tools crawl rate (slow it down) instruction.

My suggetion involves a “specific” setting in Cloudflare to block indexing on a domain. I am very sure this can be done by just setting up a page rule. However, it would be nice to see a specific easy to apply setting for this (for those not knowldegable in setting up page rules)

Thanks for listening

That sounds super cool! How would you set that up as a Page Rule?

Hi

My guess would be the only way with a page rule is to restrict something like a staging domain to your own IP only and disallow the rest. (and yeah you could block search engine user agents)

The problem with doing that though. Is needing to allow IPs like your web developer so they can access the site for work ect ect or seek opinions on the new site design from friends and other critics

However, it would be great if there was just some kind of pre set setting to stop search engines “Only” from crawling/ndexing. (because they are not always obeying instructions like robots,txt or noindex index tag) I have found. (Bing is worse)

So just a preset for stopping search engines from access/crawling would be a great shortcut tool.

If my idea sounds silly. I totally understand, Just passing on the thought.

The suggestion is more of a fast “convenience” setting/preset if anything else.

You coukd almost call it a “Bundle” setting blocking all common search engine user agents only as it adds all the common search engine user agents into the one setting instead of blocking them one by one, bingbot, googebot and the other common known search engines ect ect. A big time saver and a good help for those who dont know much about search engine user agents and other blocking methods

A good example of a problem. I put a noindex on a staging site staging .site .com and bing still indexed it while google obeyed it

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.