Activating Crawler Hints on our domain has had the unintended consequence of any GET request being submitted to IndexNow, including API calls from the client to our backend. We were not aware that this is going to happen, and didn’t initially have an appropriate ‘X-Robots-Tag’ header in place. This has lead to a bunch of responses from our API being indexed on bing (and other search engines which take their data from bing). A few questions:
- Is this the intended behaviour? It would be nice if crawler hints had more granular options, allowing, for example, subdomains being excluded from being submitted to IndexNow
- For our particular case, is setting ‘X-Robots-Tag’ on the response the best option from preventing crawlers from indexing API responses? Since we’ve made this change, we haven’t yet observed any of the previously indexed results being removed.
- We couldn’t find anything on bing’s side to allow use to remove results from their search (bing webmaster removal tool simply says something along the lines that the url is still active and the result is useful to bing users, and it refuses to do anything). Is there something to be done in this regard?