How does Crawler Hints work

[asifanwar451]

Hi,
If have a few questions about Crawler hints,

1- Is it a push or pull service from search engines point of view? Would search engines still crawl the websites?
Here’s an excerpt from the docs,

As a result, we are able to see trends in the way bots access web resources. That visibility allows us to be proactive about signaling which crawls are required vs. not

Does this mean that when a search engine bot visits a URL, CF will somehow signal whether the content changed since the last visit? For this to happen, I suppose, CF has collaboration with search engines. Do we have a list of search engines that have collaborated with Crawler hints?

2- What happens to traditional sitemaps? Will they be hidden from search engines? Do we still need sitemap and sitemap pings?

Unfortunately, CF docs have very little information about Crawler hints. Any insights would be helpful.

Cheers
Asif

[fritex]

Haven’t used a lot and tested yet, however sharing here:

How does Crawler Hints work, in detail?

cache status, some status codes, etag/last-mod. not looking at dynamic right now. cadence is still determined by crawler which uses additional signals like sitemaps, we are just providing additional heuristic to help make that decision. Working on adding more variables soon.

Early Hints feedback

Now that this is available to some it’s time to give feedback. The welcome email doesn’t provide a way to give feedback it so I will post it here. Using webpagetest with the proper command-line argument I can see the requests to some resources are initiated by the client when parsing the HTML instead of preloading from an early hint. [Capture1] Our pages have preload statements at the start of head as well as the new Link response header. The tracing shows that all resources are being preload…

Google search console crawling and cloudflare caching problem

Did you enable crawler hints, it will allow search engines to only crawl your site when you have updates rather than needlessly crawl non-updated pages https://blog.cloudflare.com/cloudflare-now-supports-indexnow/. In which case you could see total crawl requests lowered, as some pages do not have updates and crawlers won’t need to crawl them.

Maybe something changed and some new additional information could be sahre from @akrivit