We are load testing our worker implementation and found a weird behavior.
This worker internally call Cosmos to retrieve some data using @cfworker/cosmos. Everything works quite well under a light load such as 50 RPS. When the load bumped to 500 RPS, we found that all Cosmos requests are getting responses with status code 429. After inspecting the response, it is not Azure Cosmos response style and it is lack of all Azure style headers.
This issue would last for around 2-3 minutes even for a single request, then it is back to normal. We could not find any 429 response records in the Azure dashboard and we could still communicate with Azure Cosmos using other clients.
We are suspecting there might some outbound traffic rate-limiting for in Cloudflare workers. But could not find any document detailing that.
If anyone has a similar problem, could you please share more information with me?