502 Bad Gateway when calling out to a web service


#1

My worker calls out to a web service in order to decide how to handle certain downloads from my site. I can see the request coming into the web service (hosted on AWS) and (using tcpdump) I can see the response that the service sends back. The response is Ack’ed at the TCP level so I know it has been received, but my worker never successfully receives the response for some reason. Eventually it sees a 502 Bad Gateway response containing a Cloudflare error page that includes the text “Error communicating with origin server ec2-18-217-202-190.us-east-2.compute.amazonaws.com”.
If I replace the URL of the web service with the URL of an image hosted somewhere else the worker sees the response successfully, so presumably there is something about the response the web service is returning that the worker does not like. If I call the web service using Curl or a browser the response looks fine to me:

HTTP/1.1 200 Ok
Host: ec2-18-217-202-190.us-east-2.compute.amazonaws.com
Server: TcpServer
Date: Thu, 24 May 2018 15:42:26 GMT
Content-Length: 5
Content-Type: text/plain

hello

Any ideas what could be causing this, or how to diagnose further?

Many thanks


#2

I am not sure I understand. You have a webservice on Amazon that you call from a Cloudflare worker and which sends back some data that you would use on Cloudflare’s side to make a decision, right? How is that data formatted? Could it be that some header is not right (content type)?

The example you posted returns text/plain. Is that really what Cloudflare respectively your code expects? How come it works when you swap your webservice URL with an image URL? The latter would return nothing of that sort in the first place but a regular BLOB.


#3

Hello Sandro

Thanks for replying - you understand correctly, I have a webservice on AWS that I call from my worker, and I want to use the result of the service in the worker. In order to create simple PoC the webservice currently returns plain text, as in the example. The format is not really relevant because the worker never sees the response at all. It’s possible there is some header that is not right, but like I said, it looks right to me and I can call the service from other clients and get the expected response.
Here is the relevant part of my worker:

console.log("About to send PS request")
const psRequest = new Request("http://ec2-18-217-202-190.us-east-2.compute.amazonaws.com/whatever", {
  method: "GET"
})
console.log("Sent PS request")

// Wait for a response
const psResponse = await fetch(psRequest)
console.log("Got PS response, status: " + psResponse.status);
const psResponseBody = await psResponse.text()
console.log("Got PS response body")

In the debug window there is about a 60 second gap between “Sent PS request” and “Got PS response”, and the response is the 502 error I mentioned before, rather than the response sent by the webservice. If I change the URL to something different (lie an image somewhere) I see “Got PS response” immediately, as expected.
It all points to there being something about the response from my webservice that the worker does not like, but I can’t see what, and I don’t know how to diagnose it, so I’m looking for suggestions.


#4

I cant say I am all that familiar with the API :slight_smile: but from what I quickly gathered it seems to look alright.

One thing I noticed is the specified hostname is not reachable right now. Is this because you shut down that instance? If not it would explain why it hangs :slight_smile:


#5

Hello Sandro
Apologies, the instance is back up now. Remember that I said I could see the request arriving at the webservice on AWS and the response being sent back, so I know the webservice was running at the time!
Rob


#6

True, I did forget about that, my bad :slight_smile:

No need to apologise :).

Unfortunately the instance still appears to be down (http://sitemeer.com/#ec2-18-217-202-190.us-east-2.compute.amazonaws.com) so debugging is slightly difficult :slight_smile:

Maybe @cscharff would have some idea why the response seemingly does not arrive.


#7

Sandro
The firewall on the AWS instance only allows connections from Cloudflare IP addresses and my own IP addresses, so sitemeer will always report that the host is down.
Rob


#8

That would explain why my connection attempts have failed :wink:


#9

Considering you wrote you do receive an ACK from Cloudflare I’d say we can assume the firewall should not be a problem either and Cloudflare does receive the response.

Have you nonetheless tried disabling the firewall, just to be sure?


#10

I have reconfigured my AWS instance to allow inbound HTTP connections from all IP addresses and my worker is now working as expected! I really can’t explain this (given that I could previously see the response leaving my webservice and being acked), but at least I am able to make progress now. Thank you.


#11

I’ve re-run tcpdump on the AWS instance with the relaxed firewall rules in place, and I can now see why it wasn’t working before. For every request that the worker sends, I now see two requests coming into my webservice.
The first looks like this (from 172.68.58.252):

GET /whatever HTTP/1.1
Host: ec2-18-217-202-190.us-east-2.compute.amazonaws.com
Connection: Keep-Alive
Accept-Encoding: gzip
CF-RAY: 4207ce9c83992501-ORD
X-Forwarded-Proto: http
CF-Visitor: {“scheme”:“http”}
True-Client-IP: 0
CF-Connecting-IP: 2a06:98c0:3600:0:0:0:0:103
CF-EW-Via: 9a1bb7b07ec2661c11daefdfaeedf7da577272e915d02c080f5e36d5d8468f21

And the second request looks like this (from 35.202.46.36):

GET /whatever HTTP/1.1
Host: ec2-18-217-202-190.us-east-2.compute.amazonaws.com

The first IP address is a Cloudflare one which was in my original firewall rules, so I was seeing that request. The second one is a Google IP address so I wasn’t seeing that request before. Presumably the worker was relying on a response to this second request for some reason.

My question is why does the fetch command in my worker result in two HTTP requests to my webservice?

Rob


#12

It looks like this second request only happens in Preview mode.


#13

That is slightly weird, as it would mean your trigger gets executed twice. Also, where does Google come in?

Not sure what it exactly is, but you mentioned preview, which sounds like some test mode. Could that be the reason? Where does the request come from when you disable “preview”? When you send requests from your local machine, how many requests end up with Amazon?


#14

Preview mode is feature of Cloudflare’s worker editor - it lets you test your worker before deploying it. For some reason when I use this my webservice receives two requests, the second one coming from a Google IP address - no idea why.
If I save the worker and invoke it by sending a request to my origin server from an external client, then the worker works as expected.