Cache entries remain on some edge's despite purge API call

Website, Application, Performance
1

What is the name of the domain?

What is the issue you’re encountering

Despite calling the CF API to purge individual entries from the cache, they remain (outdated) in some edge servers.

What steps have you taken to resolve the issue?

Calling the API again, once the outdated cache entry is recognised, usually solves the issue.

What are the steps to reproduce the issue?

We host an APT (Debian package) server, which consists, aside of the DEB packages themselves, of compressed package list files Packages.xz as well as an InRelease file per distro/release, which contains sizes and hashes for all Packages.xz files, and an embedded signature. We cache all those files. The packages have a version string in their file name, hence their cache entry never needs to change, but the Packages.xz and InRelease files do change.

When adding or removing any package, the Packages.xz for all architectures of all APT server components which contain it, need to be rewritten, as well as the InRelease file for the APT distro/release of those components. On any update, a script scans the package directories of the distro/release to be updated, regenerates all Packages.xz files and the related InRelease, and then does a Cloudflare API call to purge those files from the Cloudflare cache:

curl -sSf 'https://api.cloudflare.com/client/v4/zones/<zone_id>/purge_cache' -H 'Authorization: Bearer <api_key>' -H 'Content-Type: application/json' --data "{\"files\":[${files%,}]}" | grep '"success":true,"errors":\[\],' || exit $?

Since we currently use the Cloudflare free plan, each API call is limited to 30 files, while in case of larger updates, more files could be affected. We hence split those into multiple calls, if needed. As can be seen, we check for curl errors and as well assure the API does not return any error. This never happens, but regularly, individual CF edge servers have individual outdated entries, and do keep them until expiry timeout or another API call to purge them.

Are there any limitations with the API, or known issues when doing multiple calls shortly after another? Since API depends on InRelease sizes and hashes to match the package list, as inherent integrity check, we need those files to be purged reliably from the cache.

2

There is also a limit of 1000 URLs per minute, is that likely to be a number you reach for a large update? (I’m not sure if/how an error is returned if some but not all of the URLs passed in a single call exceed that).

1 Like
3

Good to know, but that is above what we can reach. If really all distro/releases were updated, it would be 562 files at the moment, but e.g. the last time, the problem appeared for one of our users, it was 44 purged URLs only.

4

If cache entries are remaining on some edges despite making a purge API call, this could be due to several reasons. It might be related to the cache not being fully cleared due to improper API configuration or a delay in the cache invalidation process. Ensure that the API call is being executed correctly and that it targets all relevant cache entries. You may also want to check if the cache headers are properly set to ensure immediate cache clearing.

For apps, caching issues can cause old data to persist, affecting game performance or loading. In such cases, clearing the app’s local cache or reinstalling it might help resolve any lingering issues.

5

You can see the API call in my OP post, it works generally, is correct, headers correct, all files contained, no error returned, but "success":true,"errors":[], for each individual call.

Doing the very same call once again, solves the issue, so it is like the task is not 100% reliably forwarded to all edge servers after a call, but as it is random, doing the call once again usually forwards it to those edge servers which where missed before.

6

After we excluded package lists and signature files from the Cloudflare cache to avoid the issue, we just had a case where the same happened for OS images and their signature files. When updating downloads, we purge image files, their hash files and signature files all from the Cloudflare cache via API. Now we had cases where the signatures where successfully purged from the cache, but the image files not, so that signature verifications failed. I verified in logs that the API call really has been done, and it has. So again, somehow some calls or some files must have been lost. We are talking about 744 files overall, close to the limit, but not above. And I would assume the API to return an error, if the limit was hit? … I just tested it, cleared 2002 URLs in some seconds, and all calls returned only success and no error. Though those URLs did not exist and hence no real cache entry was cleared, it that matters.