Request: Cache Spider/Indexer



I would be interested in having a feature/function of pre filling CloudFlare’s cache rather than on-demand caching. I would be willing to pay for this per-use.

Usage Scenario
A client hosts a site from China. They are not willing to change hosting provider, but are willing to use CloudFlare. I would set up the account, and pay $x for CloudFlare to index the site and cache the resources globally (or at least somewhere in CF’s fast network, outside of China).

Every user gets a fast experience of every resource request.

One more thought… a sub option to point CF to sitemap.xml urls rather than spidering - assuming the xml files has images in additional to pages.


This has come up before…having some sort of Push/preload feature for the CDN. I don’t think Cloudflare was very receptive.

With 120+ POPs, that’s a lot of bandwidth and storage that Cloudflare would have to charge for, and most of that would be wasted for lack of use. But take a look at Argo as an example of cost: $5/month + $0.10/GB. Speaking of which, Argo kind of sounds like what you’re after.


A small part of the issue is asking a client to check out a test I prepare. My closest server may have the html and images cached - but definitely not the one where they are, since it is a private test.
I have to ask them “load the page twice to see the true speed”. Some don’t understand why, even after explaining how the CDN works. The others are dubious of the CDN doing anything since they know about browser caching. Neither scenario helps close the sale for either myself or Cloudflare.
Therefore pre caching the network is welcomed.



We do have a Prefetch URL setting available to customers on the Enterprise plan


That only effects the single server-user pair. It wouldn’t prefetch a url and cache it globally, right?


I understand the use case (not the only one who has asked for it, and asking here so folks can vote for it is great), but I’m not sure that is on our roadmap at this point. As @sdayman mentioned Argo’s tiered caching can help for this in scenarios where the concern is multiple PoPs overloading the origin for requests, but that’s not exactly what you’re looking for here.

I don’t have a great answer… in theory you could build a bot that ran in multiple cloud datacenters to make the call to their closest PoP for the assets. Seems like a lot of work and Cloudflare should build it for you darn it… heh.

If the goal is to show customers a perf difference for a site on Cloudflare vs. not… we typically do that using Catchpoint and give them a more Global or regional view depending on their needs (and I usually leave in the beginning of the test when the cache is being warmed to demonstrate even more the value of caching). Catchpoint isn’t cheap, so a tool like webpagetest can also be used, though it doesn’t give as much detail or the over time results I like to present (since anyone can cherry pick a single test).

I will say sometimes doing that type of testing does require host header overrides and/or resolve overrides which are also ENT features.


Currently when I need to demonstrate the speed difference to a client I make a hosts entry on my computer directly to the origin and capture the speed data. Then I remove the hosts entry, warm up the CF cache, and finally take more data. I then compare the two result sets.

For a live test by the client… I guess a way around the warming up step is for me to build a page on my own site that has a hidden iFrame of the page I want to demo. It would display “thank you for demoing bla bla bla, you are being redirected to the test page”. After 5 seconds (should be plenty of time) the page redirects to the live test page.