Cache not getting purged for Worker

Website, Application, Performance
1

When purging cache (happens for both Purge via Dashboard & purge via API) the request which is served by Cloudflare Worker still contains cached content (i.e. cache doesn’t get purged)
The Worker is supposed to send a prerendered version based on user agent, it does not alter any cache keys.

However, when fetching URL normally, the resource shows that the cache was purged, Cache-Status: MISS.

Example:
We purge cache for the URL: https://germantechjobs.de/jobs/Cisco/all

When we fetch the URL with user agent “googlebot” (which is served by the Worker), we get a Cache HIT with a high age.

Then, we fetch the same URL with a normal agent and it shows Cache-Status: Miss.

It looks like somehow the cache doesn’t get purged for the Worker itself.

4

There has to be something else going on in order to not serve the same cached version for googlebot as for regular visitors. How is your worker handling this?

5

We are using the official Prerender middleware for Cloudflare, which is available on:

6

It looks like they’ve changed the URL for the fetch. That’s the URL you’ll need to purge.

https://developers.cloudflare.com/workers/learning/how-the-cache-works/#single-file-purge–assets-cached-by-a-worker

7

Thanks for pointing it out!

However, the fetched URL which is built as following:
https://service.prerender.io/${url}
is not within our Cloudflare account.

So how is it possible that the worker caches a URL for us which is not within domains?
It makes absolutely 0 sense from the user perspective.

8

It’s cached in your account. It’s just using that other URL as the cache key:

Have you tried purging it?

9

This behavior doesn’t make sense from a user perspective and I don’t see any reason why Cloudflare changed it.

I also tried to purge https://service.prerender.io/https://germantechjobs.de/jobs/Cisco/all but this doesn’t purge the worker cache. Still the old, cached version.

10

Are you still having issues purging? Can you DM the url that you are trying to purge, what you have tried to purge, and your zone id

11

Yes, the issue persists:
https://devitjobs.nl/en/jobs/ServiceNow/all
(if I purge it, the URL fetched by googlebot still hits cache)

Zone ID: 56bb7c65dc4dd6e53f45030a6b44bbb0

12

The issue is with your worker, when there is a googlebot, the worker makes a fetch with a different hostname/url (https://service.prerender.io/${url}). You need to purge that hostname/url. Refer to How the Cache works · Cloudflare Workers docs for more information

1 Like
13

They tried this:

But it’s unclear if that’s the actual URL being fetched by the Worker. Maybe it’s close, but not quite right.

14

I see the problem now. The issue is with your cache rules:

Screenshot 2023-11-27 at 12.39.21 AM
Screenshot 2023-11-27 at 12.39.21 AM2150×304 29.8 KB

#9 says, bypass cache if url contains “servivce.prerender.io”

#10 overrides #9 by saying cache if user agent contains “google”

the purge request you send acts just like the fetch request your worker normally sends. When you send a purge request for https://service.prerender.io/https://germantechjobs.de/jobs/Cisco/all for example, nothing gets purged because the cache rule triggers and says cache is being bypassed so nothing would exist to be purged.

What you need to do is trigger the same features/path that the fetch/GET does so in order to purge the googlebot version, in the purge request you send, you need to include a user-agent that contains “google”

so something like:

{
  "files": [
    {
      "url": "https://service.prerender.io/https://germantechjobs.de/jobs/Cisco/all",
      "headers": {
        "User-Agent": "google"
      }
    }
  ]
}

This would trigger cache rule # 10 which says the url can be cached and therefore the purge actually happens

1 Like
15

Hi zaidoon,

I will check and let you know once it’s done.

To be honest, the default caching behavior of Cloudflare is counterintuitive to say the least.
Why does it apply all the edge cases to the cache keys, instead of using just the URL itself as the key?
There should be a setting that enables it.

I don’t see a reason why (in our Zone) the URL xyz is cached differently depending on user agents, referrers, and the worker itself. If I want to cache xyz, then I want just that and if I want to purge it, I should be able to calle purge xyz in order to purge it from all cases.

16

First of all, happy fourth anniversary of you joining the Community!

If that happened, and Google crawled your site first, then all your users will end up seeing a cached view of whatever Google got from Prerender.

For the same reason above, but in the opposite direction. If you want a special version of the cache, you’re going to have to purge the special version of the cache. If all the Prerender URLs are bad, but the rest are fine, many aren’t going to want to wipe out all the good stuff just to get rid of the bad stuff.

For non-Enterprise plans, there’s Purge URL (cache-key), and Purge Everything. Enterprise gets more options, because $$$.