Worker fetch to Cloudfront distribution resolving to distant Edge


We are using Cloudflare Workers to respond to client requests around the world. Our workers need to fetch resources (JSON) from a AWS CloudFront distribution, before returning a response to the client.

We have observed an unreliable performance, where reponse times (Time To First Byte) from the worker ranges from 50ms to 2s. Those spikes occur in various regions, at various periods of time.

So I made a simple CF Worker fetching an image stored on S3 behind CloudFront. I have disabled the CF cache, to always make a request to CloudFront (stale cache).

It appears during those spikes the fetch request did not reach the closest CloudFront edge server to the PoP. Here is a sample of those cases:

CloudFlare PoP Location           AWS CloudFront Edge Location
CPT (Cape Town, South Africa)     PER50-C1 (Perth, Australia)
CDG (Paris, France)               BOM52-C1 (Mumbai, India)
LHR (London, England)             PER50-C1 (Perth, Australia)
HKG (Hong Kong)                   VIE50-C2 (Vienna, Austria)

Just looking at requests that ran on LHR PoP, at times 40 / 120 requests were misdirected.

I have seen a topic related to resolver where such issue was apparently happening.
Is it possible the same case happens with the fetch request from the Worker ?
Is there a way to override the resolver used by fetch?