HEAD requests fail without a user-agent even though the backing site is happy to receive them


In the space of teaching and learning applications like Blackboard, Moodle, or Sakai we use a lot of iframes to pull third-party content into courses hosted on these systems. In response the X-Frame-Options features these systems started using HEAD requests to check X-Frame-Options before attempting to load the web page in the iframe and if X-Frame-Options was set force the window to be popped up in a new window so the user did not see a blank rectangle. This has been common practice for some time now.

This approach fails when the site that will be providing the “within iframe” content is behind CloudFlare. In general these HEAD requests are pretty thin on the request headers and in particular the problem that I found and diagnosed had to do with the lack of a user-agent header. Now I know that I can control the level to which CloudFlare is picky about how well formed requests need to be to pass through, but in this case, the HEAD request without a user-agent is rejected (with a 403 actually).

Here is the bug (and fix in Sakai):


Now I worked around this in Sakai but that fix won’t be in production around the world for 6-12 months and lots of other learning systems like Blackboard may not even know about this and their lead time to fix things is also 6-12 months once they notice this issue.

I would love to be able to keep these learning tool and learning content sites behind CloudFlare but this inability to have a proper HEAD request is kind of a deal breaker.

I want to encourage everyone that builds these kinds of learning tools to use CloudFlare because it is just amazing in terms of DDOS, “under attack”, “free-location-tagging” etc! But we need iframes and HEAD to work.

Thanks in advance.