Worker having relatively long connect time


I have a really simple worker that has almost no cpu time (0.3 ms median) but it seems to have a relatively long connect time. One of my client sent me logs that shows that it usually take 500ms or more to connect to this endpoint. The problem is : 500 ms is too much for our needs.
His servers are on AWS germany and, from what I found, a cloudflare worker should be accessible everywhere-ish in less than 50ms.
Here is an example of such log after a timeout of 1 second on his part :

"transfer_time": 1001.0,
"handler_stats": {
"url": "my_worker_url",
"content_type": null,
"http_code": 0,
"header_size": 0,
"request_size": 710,
"filetime": -1,
"ssl_verify_result": 0,
"redirect_count": 0,
"total_time": 1.000986,
"namelookup_time": 0.010149,
"connect_time": 0.258574,
"pretransfer_time": 0.526432,
"size_upload": 544.0,
"size_download": 0.0,
"speed_download": 0.0,
"speed_upload": 543.0,
"download_content_length": -1.0,
"upload_content_length": 544.0,
"starttransfer_time": 0.0,
"redirect_time": 0.0,
"redirect_url": "",
"primary_ip": "ip_address",
"certinfo": [],
"primary_port": 443,
"local_ip": "ip_address",
"local_port": port,
"http_version": 0,
"protocol": 2,
"ssl_verifyresult": 0,
"scheme": "HTTPS",
"appconnect_time_us": 526389,
"connect_time_us": 258574,
"namelookup_time_us": 10149,
"pretransfer_time_us": 526432,
"redirect_time_us": 0,
"starttransfer_time_us": 0,
"total_time_us": 1000986,
"appconnect_time": 0.526389

I’m not an expert on network and TCP protocol so sorry if my question is not properly written but feel free to ask questions if you need more info.
How is that possible for such a simple worker to timeout after 1 second ?