Cloudflare generates random URL links from my site, and made them available to Bing and Yandex for indexing. How do I stop Cloudflare indexing? Please let me know.
Can you share an example of these random URLs Cloudflare is generating? That sounds very odd.
In addition, do you have crawler hints enabled?
Yes, crawler hints was enabled. But 2 days back i disabled as I thought it may cause this issue.
Seems like your server returns “200 OK” on any URL starting with https://www.tamilsonline.com/privacy.aspx/ for example:
$ curl -I 'https://www.tamilsonline.com/privacy.aspx/aaaaaaaaaaaaaaaaaaaaaaaaa'
HTTP/2 200
So basically anyone anywhere in the world can make a request like that, Cloudflare will see the 200 OK, and (if crawler hints is turned on) they’ll send the URL off to Bing and Yandex. So it’s not Cloudflare generating random URLs, it could literally be anybody. And Cloudflare just observes your server saying “YES THIS EXISTS” and does its thing.
Your server should NOT be responding to these requests with a 200 OK, or at the very least you should include an x-robots-tag in the response to ask search engines not to index the page.
I realize that “privacy.aspx” is some kind of script, it’s probably treating the rest of the URL as input (and then doing nothing with it). But you need to do something about it.
Modifying the script would be one option, but you could also try to catch requests to “privacy.aspx/*” and either redirect them or blackhole them. You could do this either on your origin server or through Cloudflare.
1 Like
Thank you for your suggestions, I will try and let you know the outcome later.
1 Like
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.