Custom Purge not working with Pages

[skorov]

Hey team!

I’m trying to remove a particular file from my website. The file in question is located here:

https://handbook.volkis.com.au/redirects.json

The site lives in Cloudflare Pages and using Custom Purge is not working. Here’s what I have done so far:

• Checked to make sure the underlying hostname does not still contain the file. It doesn’t.

> curl -I -H 'Cache-Control: no-cache' https://handbook-volkis.pages.dev/redirects.json
HTTP/1.1 404 Not Found
...

• Checked the Age header of the cached file response. It does not reset after Custom Purge. The value is currently 60089
• Rebuilt the Pages environment, then Custom Purge again. Nope
• Waited 8+ hours overnight for the cache to simply expire. It hasn’t.
• Turned on Developer Mode. I could still see the file.
• Set the Browser Cache TTL to 2 minutes. Grasping as straws here…

Any assistance would be really appreciated.

[MarkusHeinemann]

Hello,

have you cleared the browser cache. For me the file is unavailable:

[skorov]

Well that’s interesting! I definitely checked that, even on other machines that have never seen that file before.

If it helps, I’m in Sydney, Australia. Maybe that matters and the cache is not being delete on all edge servers?

;; ANSWER SECTION:
handbook.volkis.com.au. 300     IN      A       172.66.43.173
handbook.volkis.com.au. 300     IN      A       172.66.40.83

[MarkusHeinemann]

That is strange. The tool locabrowser.com receives a 404 form the canada location as well (i’m located in germany).

[skorov]

Using Locabrowser, it seems like only Australia is having this issue…

Edit: Additional info shows that Australia is being routed to an edge whose cache is not cleared:

Edit 2: Wanted to add more info to help CF staff track down the problem. Here are the response headers:

HTTP/1.1 200 OK
Date: Thu, 02 Dec 2021 02:20:59 GMT
Content-Type: application/json
Connection: keep-alive
Age: 73608
Cache-Control: public, s-maxage=604800
ETag: W/"66efd6b61ff3ec0b10a4ba663c7d10b1"
CF-Cache-Status: DYNAMIC
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
x-robots-tag: noindex
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=xEguZc4IeUEcUMdM8MmzFUMZD9wc56ZCYllHJfEilsHnbvWFL%2Fvu5GPhxvvAQD5pUGSRyM7IsVO54guC3ZwJVJrCnoZ9D%2BeXH0lhIA9UrCHvea1nTMjTMhaC%2F%2B7MsYwxOQOW6Ln7f6QC"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Vary: Accept-Encoding
Server: cloudflare
CF-RAY: 6b711444dfaf5503-SYD
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400

[skorov]

I thought I’d come back and update this post since I’ve received some helpful information from the Cloudflare team.

According to their engineering team:

There are two layers of caching with a site built on Pages using a custom domain. (ie - mysite.com → mysite.pages.dev) The cache the you control on the custom domain (mysite.com) and what Pages does. So although you purged the cache on your zone, Pages does have a cache retention policy of roughly one week for any assets. This was an intentional decision made to help mitigate any issues client-side when a user deploys a new version of their site.

Currently we have verified that the asset is no longer served from cache.

My main concern with this is that it becomes impossible to purge sensitive information that might have been published by accident. That sensitive information would hang around for 7 days after being removed. Not ideal!

Edit: The Pages team came back with this.

Regarding the sensitive data issue purging, they said:

We are aware of this concern and are working for a solution related to Pages Caching improvements or a config to handle this better

Since the Pages cache only applies to old, deleted files, a workaround I’ve found would be to:

1. Overwrite the data containing sensitive data with an empty file of the same name.
2. Wait for build.
3. Delete the now-empty file.