Bad gateway 502 for sites on different subnet

Zero Trust Cloudflare Tunnel
1

I see error 502 is regularly posted, though I have not seen any posts with my specific situation.

I have a network which has multiple separate networks connected by a router, each network has it’s own IP range. They are all accessible from the main network, and the Cloudflared machine is able to reach all subnets as well.

However, hostnames will only work for sites within my main network, 172.16.100.x/24, every site I have tried so far within this subnet works fine.

I now want to add hostnames to sites in the other networks, for example 172.16.102.0/24, but ALL sites in non-default subnets throw the 502 Bad Gateway error. Even a simple http server shows this error.

Is there some special configuration required to allow cloudflared to access other subnets?

Here are my logs:

Aug 17 22:58:31 cloudflared cloudflared[1935]: 2023-08-17T22:58:31Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from >
Aug 17 22:58:31 cloudflared cloudflared[1935]: 2023-08-17T22:58:31Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to>
Aug 17 22:58:36 cloudflared cloudflared[1935]: 2023-08-17T22:58:36Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from >
Aug 17 22:58:36 cloudflared cloudflared[1935]: 2023-08-17T22:58:36Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to>
Aug 17 22:58:36 cloudflared cloudflared[1935]: 2023-08-17T22:58:36Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from >
Aug 17 22:58:36 cloudflared cloudflared[1935]: 2023-08-17T22:58:36Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to>
4

Just realised the above logs got truncated, here is a full line:

Aug 17 22:58:36 cloudflared cloudflared[1935]: 2023-08-17T22:58:36Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: server selected unsupported protocol version 301" connIndex=0 dest=https://[hostname]/ event=0 ip=198.41.192.37 type=http

(I’ve sanitized the hostname above)

5

The above log line relates to a HTTPS service (with TLS checking disabled).
Here is a log from the same device via HTTP:

Aug 17 23:34:49 cloudflared cloudflared[1935]: 2023-08-17T23:34:49Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: EOF" connIndex=1 dest=https://[hostname]/favicon.ico event=0 ip=198.41.200.193 type=http
6

I think I may be onto something here, and believe it could be related to the MTU.

The network with this subnet has a limited MTU, known to be 1484. However, when pinging an endpoint with a specific MTU, I had to go as low as 1460 before a ping was successful.

I have tried setting my cloudflared machine’s physical MTU to various lower values, even down to 1452 and have been successful in connecting to one particular host which wasn’t previously working. However, it’s not the host that I need to get working, which still shows the previous EOF error in the logs.

Not sure what to check next, could someone with more experience shed any light?

7

As long as you can access the other subnet from the same host, then cloudflared should be able to. Can you curl the host from the server running cloudflared?

8

Thanks Cyb3r-Jak3

Here is the curl output:from the device we would like to setup:

robf@cloudflared:~$ curl -I --insecure http://172.16.102.2
HTTP/1.0 200 OK
Server: GoAhead-Webs
Pragma: no-cache
Cache-control: no-cache
Content-Type: text/xml;charset=ISO-8859-1
Content-length: 345
Date: Fri, 18 Aug 2023 06:48:10 GMT

And this is from another device which after the MTU adjustment now works (the curl output before and after MTU adjustment is the same):
Starts with a http redirect:

robf@cloudflared:~$ curl -I --insecure http://172.16.102.3/INDEX.htm
HTTP/1.1 200 OK
Expires: Mon, 31 DEC 2035 12:00:00 GMT
MIME-version: 1.0
Content-Type:text/html
Content-length: 1066
Content-Encoding: gzip
9

Are you using https or http? Because if you are using http then the tunnel config will need to match. Can you share your tunnel config?

10

This host we want to use HTTP. It does support HTTPS as well, but uses a weird outdated cipher which browsers don’t support, so HTTP should always work for this. It’s basically a network device with a web configuration interface

11

Is your tunnel configured for HTTP or HTTPS to the origin?