Intermittent incredibly slow image loading

bug

#1

I just started getting a problem where image files will randomly take a really really long time to load. I’ll load a page and it will load instantly, then I’ll ctrl+F5 a couple times and it will be fine, but then randomly one of the images will take minutes for TTFB. When it finally starts loading it takes milliseconds to get the file. The waterfall looks like this:

I’m fairly sure this is a cloudflare issue because I’m getting the same problem on two different servers that are in two physically different data centers running two different versions of CentOS. Both are on 1gbps ports and under minimal load.

I don’t want to turn off cloudflare for testing because I’ve had some bad DDoS experiences and I don’t want to risk exposing my origin IP.


#2

Do you have any DDOS protections or rate limiting systems on your own server? If this happens after some reloads, some system might be flagging and/or throttling the connection. Make sure to whitelist Cloudflare IP addresses and use the X-Forwarded-For header for the visitor’s real IP.


#3

I have only a minimal set of apache modules enabled, with the cloudflare module to show users real IP.
iptables is configured to only allow connections from cloudflare servers.

The problem also occurs when loading new pages. It seems like whenever cloudflare caches an image it just downloads it really slowly. Sometimes I can even see in the apache server-status page that the “Req” field is over 60000 for some image requests. Downloading a large file (which cloudflare does not cache or mess with) from my server starts instantly and proceeds at 6MB/s (max for my home connection).

I bit the bullet and bypassed couldflare and stopped iptables for testing and the problem went away immediately. It came back when I re-enabled cloudflare.