Specifics Behind IndexNow & Cloudflare

I have questions regarding exactly what triggers a send to search engines participating in IndexNow, and what exactly is sent when that trigger is fired.

I know IndexNow is about telling search engines what “pages” have changed and feeding those pages to the crawlers.

But what exactly qualifies as a “change?” If a script or API location/URL changes, does that trigger IndexNow? Any more detail around this will be very helpful.

More importantly, for the website and situation I’m working with, what exactly is Cloudflare sending to the crawlers? Is it the HTML source document from HTML pages that have been requested and have some form of alteration in comparison to the last request?

If a JS file executed by the browser on a specific page or pages changes its output and thus the fully rendered content from one day to the next, does that trigger a crawl?

Given the current state of the web and abundance of client-side heavy websites/applications, would any change to the fully rendered version of a page trigger IndexNow, or does it need to be a chance in the source/initial HTML response and not the rendered version of a page?

I’ve probably explained my thoughts and questions like an idiot, but ultimately I’m asking Cloudflare to pop the hood on how IndexNow works from a technical standpoint and if that happens, I will be able to answer my own autistically-written questions :slight_smile:

1 Like

A list of the URLs that have changed. It is up to the participating search engine to then fetch the URL and reindex the pages. Apparently CF will keep notes on which search engine has indexed what pages and when, and only notify them if they know the search engine has not indexed a page since the page was last updated.

From the first blog post, it is a hash of the file or the last modified timestamp. So any detected change will trigger IndexNow. This is probably not the most efficient outcome, but it would be difficult for Cloudflare to automatically detect that the change was to the substance of the page, rather than a cosmetic or invisible change. Just remember that this is a hint, not an instruction, and search engines are free to do whatever they want with the information. Personally, I’d prefer an API that I could ping with the URLs that needed to be Indexed. (If it could do a purge/revalidation at the same time that would be nice as well)

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.