Understanding Uncached Requests


#1

Disclaimer: I’ve managed web servers hands on from the terminal since 1995, but I am a complete newbie at CDN’s and CF is new. So excuse my ignorant queries.

I got a pitch from CF sales campaign for Argo. But we usually want to understand the context and what’s happening before making a business decision to spend money, even if it appears to be “cheap” it will add up over time… and this uncached traffic thing is a 95% opaque.

the email campaign analyzed our site with this result

Total requests: 7,050,022
Uncached requests: 4,844,894
Total bandwidth: 634.51 Gigabytes
Total uncached bandwidth: 392.44 Gigabytes

In our world, before making any decisions… it all about logs!

  1. What exactly are these uncached requests?
  2. what files?
  3. when?
  4. Are some files cached and then later served uncached?
  5. Are some files uncached and later cached?
  6. Over what time span does either 4 or 5 above happen and why?
  7. uncached from all CF CDN servers in all countries? or only some countries?
  8. Is it about mimetypes? (file extensions)?
  9. What about our heavy media files: PDFs, Mobi, ePub and MP3 files? Are they cached, uncached?
  10. Are the uncached request to our blog, because of our page rules? (WordPress + CF was so buggy we turned off caching for the blog until our R & D figures out what the H… is going on with WP it was a sink hole of time and energy…)

Seems CF is fishing for us to buy in to services, which is OK, it’s a great service and they need a revenue stream… I get that… no problem there… but really, to make an informed decision, I need to see exactly what is happening.


#2

In our world, before making any decisions… it all about logs!

Information about the requests from Cloudflare should be in your logs (Apache or NGINX) of the webserver. Any request to your origin would be an uncached request.

  1. What exactly are these uncached requests?

Requests where Cloudflare has to go to the origin for content either because it is not cached content or because the cached items has expired and needs to be revalidated.

  1. what files?

Files which aren’t currently cached, files which are cached but need to be revalidated, content which isn’t a cached file type (when cache everything is not enabled), dynamic calls for links like database results for a search or product listings.

  1. when?

When that asset or request is made by a browser and the content is available in the Cloudflare cache.

  1. Are some files cached and then later served uncached?

It depends, expired content or content which is infrequently accessed can be ejected from the cache.

  1. Are some files uncached and later cached?

It depends on your settings, but all content is initially uncached as Cloudflare is a pull cache, so we won’t store content on our edge until a user requests it and then it is done on a per POP basis.

  1. Over what time span does either 4 or 5 above happen and why?

4 is generally based on the TTL for an object, but if a piece of content is given a long TTL (say 30 days) but not accessed frequently it is likely that after a couple of weeks that object would be removed form the cache. If it is being accessed frequently it would continue to be served fromt he cache through the lifetime of the TTL.

  1. uncached from all CF CDN servers in all countries? or only some countries?

Uncached at the POP for the user who requested the content.

  1. Is it about mimetypes? (file extensions)?
  1. What about our heavy media files: PDFs, Mobi, ePub and MP3 files? Are they cached, uncached?

IBID

  1. Are the uncached request to our blog, because of our page rules? (WordPress + CF was so buggy we turned off caching for the blog until our R & D figures out what the H… is going on with WP it was a sink hole of time and energy…)

It would depend on your page rules. If you’ve disabled caching for the blog then we wouldn’t be caching so those would generally be uncached requests.


#3

Charff:

Sorry for delay in responding… got deep into edtorial/content dev for a couple of weeks.

Thank you for this very thorough response.

bottom Line I think, since we have no page rules that would block content from being cached, is that you are not handling mp3’s which comprise a lot of our content. I will check the Apache logs and review there… of course but I’m pretty sure we are facing the music on audio files. I don’t know how Argo would help us with that… looks like a file type requiring a paid service.

We have a mobile app whose most popular feature is an audio/player module that streams MP3’s from the server and it’s kind of like our own “Spotify” for our own audio library which is extensive with music, songs, “talking head” audio etc. Users can also choose to download those files to their phone for off line listening. I’m getting hits on this module from peopel all over world. Right now we are not promoting the app heavily as the current “live” version is a kind of beta test. We really had to let it fly in order to surface issues in the Android wild,wild west.

We will also be serving “modules” think of it line “level2.zip” for a given edutainment package… which will not fit in the initial app package, so users can chose to download these “content plug-ins” and some will be quite big (30-50MB) We are still under <1000 users, but it’s climbing daily and I’m “afraid” of success… if it goes viral…

Once we fix a few bugs and upgrade a few modules we will start promoting heavily. IF we get 2000+ users hitting the mp3’s our little box on Linode will start to struggle. I’ll need to evaluate your caching service (paid) vs upgrading the linode cloud instance. certainly CF would do a better job…

More than you needed to know, but… anyway… but just in case you have some comments… appreciate your thorough attention my questions.

I see if I install Clair I can also tell if a page is cached or not… which should be interesting.

Brahmanathaswami


#4

OK I spent time searching CF site but don’t find an answer to “how much extra will it cost to cache:”

*.mp3 # 5mb - 80mb sound files
*.zip # 30mb-100mb packages

?

Your plans page:

https://www.cloudflare.com/plans/

makes no mention of additional file types / heavy media / zipped packages


#5

There’s no extra charge, that’s just not a default cached content type. You can use page rules to force it to be cached either by using mysite.com/cotent/mp3directory set to cache everything. Or perhaps by using mysite.com/*.mp3 set to cache everything. I think both should work, though I haven’t tested the second one to be 100% sure.