Disclaimer: I’ve managed web servers hands on from the terminal since 1995, but I am a complete newbie at CDN’s and CF is new. So excuse my ignorant queries.
I got a pitch from CF sales campaign for Argo. But we usually want to understand the context and what’s happening before making a business decision to spend money, even if it appears to be “cheap” it will add up over time… and this uncached traffic thing is a 95% opaque.
the email campaign analyzed our site with this result
Total requests: 7,050,022
Uncached requests: 4,844,894
Total bandwidth: 634.51 Gigabytes
Total uncached bandwidth: 392.44 Gigabytes
In our world, before making any decisions… it all about logs!
What exactly are these uncached requests?
what files?
when?
Are some files cached and then later served uncached?
Are some files uncached and later cached?
Over what time span does either 4 or 5 above happen and why?
uncached from all CF CDN servers in all countries? or only some countries?
Is it about mimetypes? (file extensions)?
What about our heavy media files: PDFs, Mobi, ePub and MP3 files? Are they cached, uncached?
Are the uncached request to our blog, because of our page rules? (WordPress + CF was so buggy we turned off caching for the blog until our R & D figures out what the H… is going on with WP it was a sink hole of time and energy…)
Seems CF is fishing for us to buy in to services, which is OK, it’s a great service and they need a revenue stream… I get that… no problem there… but really, to make an informed decision, I need to see exactly what is happening.
In our world, before making any decisions… it all about logs!
Information about the requests from Cloudflare should be in your logs (Apache or NGINX) of the webserver. Any request to your origin would be an uncached request.
What exactly are these uncached requests?
Requests where Cloudflare has to go to the origin for content either because it is not cached content or because the cached items has expired and needs to be revalidated.
what files?
Files which aren’t currently cached, files which are cached but need to be revalidated, content which isn’t a cached file type (when cache everything is not enabled), dynamic calls for links like database results for a search or product listings.
when?
When that asset or request is made by a browser and the content is available in the Cloudflare cache.
Are some files cached and then later served uncached?
It depends, expired content or content which is infrequently accessed can be ejected from the cache.
Are some files uncached and later cached?
It depends on your settings, but all content is initially uncached as Cloudflare is a pull cache, so we won’t store content on our edge until a user requests it and then it is done on a per POP basis.
Over what time span does either 4 or 5 above happen and why?
4 is generally based on the TTL for an object, but if a piece of content is given a long TTL (say 30 days) but not accessed frequently it is likely that after a couple of weeks that object would be removed form the cache. If it is being accessed frequently it would continue to be served fromt he cache through the lifetime of the TTL.
uncached from all CF CDN servers in all countries? or only some countries?
Uncached at the POP for the user who requested the content.
What about our heavy media files: PDFs, Mobi, ePub and MP3 files? Are they cached, uncached?
IBID
Are the uncached request to our blog, because of our page rules? (WordPress + CF was so buggy we turned off caching for the blog until our R & D figures out what the H… is going on with WP it was a sink hole of time and energy…)
It would depend on your page rules. If you’ve disabled caching for the blog then we wouldn’t be caching so those would generally be uncached requests.
Sorry for delay in responding… got deep into edtorial/content dev for a couple of weeks.
Thank you for this very thorough response.
bottom Line I think, since we have no page rules that would block content from being cached, is that you are not handling mp3’s which comprise a lot of our content. I will check the Apache logs and review there… of course but I’m pretty sure we are facing the music on audio files. I don’t know how Argo would help us with that… looks like a file type requiring a paid service.
We have a mobile app whose most popular feature is an audio/player module that streams MP3’s from the server and it’s kind of like our own “Spotify” for our own audio library which is extensive with music, songs, “talking head” audio etc. Users can also choose to download those files to their phone for off line listening. I’m getting hits on this module from peopel all over world. Right now we are not promoting the app heavily as the current “live” version is a kind of beta test. We really had to let it fly in order to surface issues in the Android wild,wild west.
We will also be serving “modules” think of it line “level2.zip” for a given edutainment package… which will not fit in the initial app package, so users can chose to download these “content plug-ins” and some will be quite big (30-50MB) We are still under <1000 users, but it’s climbing daily and I’m “afraid” of success… if it goes viral…
Once we fix a few bugs and upgrade a few modules we will start promoting heavily. IF we get 2000+ users hitting the mp3’s our little box on Linode will start to struggle. I’ll need to evaluate your caching service (paid) vs upgrading the linode cloud instance. certainly CF would do a better job…
More than you needed to know, but… anyway… but just in case you have some comments… appreciate your thorough attention my questions.
I see if I install Clair I can also tell if a page is cached or not… which should be interesting.
There’s no extra charge, that’s just not a default cached content type. You can use page rules to force it to be cached either by using mysite.com/cotent/mp3directory set to cache everything. Or perhaps by using mysite.com/*.mp3 set to cache everything. I think both should work, though I haven’t tested the second one to be 100% sure.
Hello,
I have a new website hosted at the free Cloudflare program (blackbeetledesign.com) and I have an issue with the cached elements.
Although it’s a static website, the percentage of cached was around 50% at the beginning.
Then I have decided to make use of the Cache Everything page rule 2 days ago and since then the Uncached went up to 90%, which is exactly the opposite of what I wanted to achieve.
Well, thanks for checking,.but I am trying to understand what is happening with the stats.
First, of all what did you check exactly? Just the homepage? From my tests, the homepage is still marked as a MISS for the cf-cache-status.
The same happens if I check some of the internal pages. Shouldn’t some of them return as HIT?
Or else what is the point of activating the Cache Everything page rule?
The response should be MISS only for the first request from CF-RAY. The next requests are HIT because CF-RAY already cached it. Then you will see HIT only if you refresh the page after the first loading (if nobody requested the page before you using same CF-RAY).
If i get the whole concept right, if someone visits from my location the website, then for the first visit it would be MISS, but then after the page gets cached on Cloudflare, all next visits by ALL USERS should be as HIT.
Is this correct?
I just checked 3-4 main subpages and all of them appear as MISS, which doesn’t make sense as I have been visiting them every day and they should have been cached by now.
Moreover, I am checking the Cloudflare stats through the siteground site tools and 1/3 of my requests are still Uncached for the last 24hours. I think the Everything cache page rule doesn’t work correctly.
Also which tool is recommended for checking the CF-cache-status? I am using the redbot.org page and also the Dev tools/network info of Chrome.