Certain URLs ending with xml.gz on my website are returning this error: “406 - Client browser does not accept the MIME type of the requested page.”
If I visit the URLs without going through Cloudflare by using a hosts.ini file modification, then I don’t get the error.
I have triple checked that the files are on the server (Windows + IIS). It’s just that Cloudflare isn’t delivering them correctly for me (and anyone else who visits the URLs). These are gzipped site maps, so it’s important to me that Google can fetch them; right now, it cannot.
This error is new, by the way. I never had it before. I’ve made no changes to the files or to the servers. Did something change at Cloudflare?
OK, well it seems that if I purge these error-inducing files from my Cloudflare cache and wait a minute, then the error goes away. I still have no idea why this would be happening but at least I’ve found a short-term solution. It’s a bit scary that Cloudflare is throwing random 406 errors when it shouldn’t. I happened to catch these, but makes me wonder how often does this kind of thing happen and I just lose the visitor.
Maybe setup a cache rule to bypass the cache on the path so that the files arent getting cached as that might be the error. You can do this in your cache rules in the Cloudflare dashboard. Here is an example.
If you use Google Adsense you absolutely want to bypass cache for the ads.txt file. It’s mime-type flubs on a lot of different bot visits and once a bot triggers a 406 you do not want that 406 to propegate.
Google will claw back your earnings if its not promptly fixed.
The 406 doesn’t happen to ads.txt with cloudflare unless you use the cache everything option.
Personally, since you don’t want admin files ever caching, just ad ads.txt to the list of config files that cannot be cached.
Also, instead of relying on only cache bypass, use your htaccess file to set the origin cache length on those files to something very short, like 3 minutes. Cache everything will obey.
