Crawler Hints - extremely dangerous


I would like to notify that Crawler Hints (IndexNow Bing) is extremely hazardous and could lead to the exposure of confidential documents. Documents that should never be accessible to third parties, and are typically found via long hashed URLs, for example, websitecom/storage/77d1fe4b880e2af0922e556392aa45953fc2a308.pdf, are now being indexed by Bing, compromising the privacy of all such documents.

These documents are usually accessible only to a specific group of individuals after logging into the administrator panel. However, if Crawler Hints is activated, a request to index these documents is sent to Bing.

Yeah, totally get your point about Cloudflare’s crawler hints and privacy. It’s a real drag on crawl budget as well, especially when sites get blasted with scans and all that data gets thrown at search engines like Bing, Yandex, Naver, and so on.

But hey, just a heads up, this isn’t really on IndexNow. If you handle IndexNow yourself, like manually integrating it or using a WordPress plugin like Yoast or RankMath, it’s pretty solid. The real issue is with Cloudflare, not IndexNow itself.

Are you still encountering this issue? If so, can you share your robots.txt file?

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.