Prefetch URLs and caching not working

I am running a multisite website in wordpress together with wp-rocket and wpml and cloudflare enterprise (Apo plugin).

Performance is that said poor. Most pages requested return uncached. Example:

Cf-Apo-Via: origin,miss
Cf-CacheStatus: Expired

I have added prefetching (modified response header in Cloudlflare) Link: https://fxscouts.com/manifest-a.txt; rel=“prefetch”;
https://fxscouts.com/manifest-b.txt; rel=“prefetch”

But URL’s still don’t get cached until after the first visit. Does anyone have any ideas?

I would have expected cloudflare to have cached the urls, even if caches have been cleared I would have expected cloudflare to have started re-caching the pages, however this does not seem to be the case.

Thanks

That is correct, Cloudflare only caches pages after they have been visited.

Cloudflare does not proactively crawl your site to cache it.

That said Cloudflare can prefetch (read cache) URL’s according to Prefetch URLs · Cloudflare Speed docs. Provided that the URL’s to prefetch has been defined in the response header.

In my case I have done this, which to my understanding should cache the URL’s before they have been visited, or am I wrong?

To my knowledge, files are still only prefetched after a first request, not automatically after a cache purge.

Also, none of the files in your prefetch manifest seem to be cacheable by default, and thus are not eligible for prefetch.

The Dashboard seems to be having problems right now, but I’ll try if it works as explained in the docs when it’s working again.

Dashboard is available again now.

It works as described in the docs. After an initial request, files with endings that are cacheable by default are prefetched, while files without endings or .html endings are not prefetched.

Thank you for clarifying. What you write does make sense.

Question: To your best knowledge does Cloudflare offer any functionality where URL’s do get cached before they have been requested?

On our system we have cron jobs which update data and properties on the webpages and thus caches are purged. Of course we could say lets only purge once every 24 hours, however this would be a sub optimal user experience, hence as to why I would like to “re-cache” URLs.

Does this make sense to you, and if so do you know of any solution or can you point me in the right direction?

If you multiply the 640 pages in your manifest by Cloudflare’s >300 locations, you’d end up with ~200.000 requests, each taking ~5 seconds to generate the site. Even with tiered Caching, that would be absolutely ridiculous.

I would instead look into a local cache (like WP Rocket) that can preload the cache after a purge. That, together with tiered Cache, should drastically reduce the load on your server and response times.

You could then increase performance even more by not purging all files at once, but one after another.

That seems much more reasonable to me than having Cloudflare effectively DDoS your server every time you purge your Cache.

Thank you for the insightful analysis and suggestions. You’re absolutely correct that a full Cloudflare purge with that many pages could lead to a heavy spike on the server.

We do currently utilize WP-Rocket for cache preloading after purging. Unfortunately, despite this optimization, we’re still experiencing longer than desirable server response times. This indicates there may be a deeper issue we need to address.

You could look into Cache Reserve. That way, Cloudflare would only request every page once from your server.

The first request would still take as long as it does (probably), but subsequent requests from different locations would load the file from Cloudflare, not from your origin.

Other than that, it’s really hard to give recommendations without knowing any specifics about your setup and testing where exactly the long response times originate.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.