My worker calls out to a web service in order to decide how to handle certain downloads from my site. I can see the request coming into the web service (hosted on AWS) and (using tcpdump) I can see the response that the service sends back. The response is Ack’ed at the TCP level so I know it has been received, but my worker never successfully receives the response for some reason. Eventually it sees a 502 Bad Gateway response containing a Cloudflare error page that includes the text “Error communicating with origin server ec2-18-217-202-190.us-east-2.compute.amazonaws.com”.
If I replace the URL of the web service with the URL of an image hosted somewhere else the worker sees the response successfully, so presumably there is something about the response the web service is returning that the worker does not like. If I call the web service using Curl or a browser the response looks fine to me:
HTTP/1.1 200 Ok
Date: Thu, 24 May 2018 15:42:26 GMT
Any ideas what could be causing this, or how to diagnose further?