Crawler Hints & 404 pages

I know Cloudflare has a lot of genius engineers. Can’t they come up with an algorithm that prevents 404 permalinks/pages from being submitted in Crawler Hints?

One scrapper bot trying to open a page that didn’t exist anywhere on your website, then it will be 404, and then suddenly other legit search engine spiders are also crawling that 404 page.

It becomes an irony when the purpose of Crawler Hints is said to make SE spiders crawl more efficiently and effectively by providing high quality data, but instead, they actually participate in crawling a lot of imaginative pages (404s) generated by other bots.

I was also thinking about this and got here while searching the forum. Several 404 pages of my personal, Cloudflare Pages hosted, website are here and there on internet in addition to a few more issues that need a different thread.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.