Testing connection directly to Azure. Trying different cache options.
What are the steps to reproduce the issue?
Requests going to Azure through cloudflare appear to be taking 20-30 seconds. If we do a request using postman direct to Azure the requests take milliseconds.
That will show us the COLO and IP that you are connecting to. Once you have that can you please run an MTR to that IP from your machine to determine no network issues are present.
Once that is done can you run a request like that again and then send us the headers with any sensitive information redacted? So we can see if this is a dynamic request or cache HIT?
Look forward to hearing from you with this information!
Thank you for coming back to us and offering to help.
After a lot of trial and error we appear to have fixed the issue.
What we did:
disabled http/2 to the origin server.
What was wrong:
Our Origin server is actually a “scalable cloud service”, it uses a load balancer in front of a pool of web servers. The load balancer that comes with our “cloud service” is pure TCP. So when http/2 to the origin was turned on it tried to send all the traffic over 1 socket to just 1 server in our pool of servers.
We didn’t spot this straight away as at the same time as http/2 being turned on we started getting errors due to some of our JSON bodys (POST and PUT) being truncated at the 48KiB (49152 byte) mark.
This was causing the load balancer to select a new server which was somewhat mitigating the problem. I presume this was closing the http/2 connection and then Cloudflare was creating a new http/2 connection which would go to another server.
We worked around the 48KiB limit and then all clients stopped responding as all requests were being delayed by 15 to 30s.
Now that http/2 is turned off traffic has returned to normal.
