Cloudflare Tunnel - 502 Error

Background: I have Cloudflared running in a Docker container along with several other aggregator containers…and it was working until yesterday!

Below is the snippet from my docker compose. Token is contained in an .env file seperate.

I was previously using similar to the below
Cloudflare->Zero Trust->Tunnels
Tunnel Name = “mytunnel”, Status “Healthy”

Public Hostname
Subdomain = mycontainer
Domain = mydomain
Service = HTTP
URL = 192.168.0.25:5055

Within my docker compose, I run all my containers using Portainer within a single stack. I’ve declared the cloudflared container to my stack (so it’s running in the same stack on the same docker proxy network) as below.

services:

Cloudflare tunnel

tunnel:
container_name: cloudflared
image: cloudflare/cloudflared
restart: unless-stopped
environment:
- TUNNEL_TOKEN=${CLOUDFLARE_TOKEN}
command: tunnel --no-autoupdate run

I have this all running on a Terromaster F4 NAS with latest TOS version. I can see the container listening on docker-proxy, on the appropriate port. The NAS IP is reserved. Checking the “Cloudflared” log, via Portainer logs, I can see the following:

2023-12-18T18:36:21Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp 192.168.0.25:5055: i/o timeout" connIndex=3 dest=https://mycontainer.mydomain.online/sw.js event=0 ip=198.41.192.107 type=http

I’ve recreated the tunnel in Cloudflare. I’ve refresh’d the token several times. Redeployed the stack. Repulled the image.

My thinking…since I’m seeing attempts to reach the container being logged…the tunnel is making it into my NAS. When I go to the IP above, locally on my network, the requested server is reachable. Something between that service (my origin) and Cloudflared container seems to be disconnecting (or timing out).

I’ve read a TONNE of the posts similar to these (502 Error)…yet I don’t see any solutions that have worked for me.

Given the lightweight nature of the cloudflared container…I know there isn’t any console to connect to…is there any other troubleshooting tasks/items/steps that someone could point me in the direction of?? Really perplexed as to how and why this stopped without any real reason (that I can see).

Thanks in advance!

Can you curl 192.168.0.25:5055 from the machine running the docker container? Maybe cloudflared triggered a block rule on your NAS?

Thanks for the suggestion!

Logging into ssh on the NAS, ter_curl (curl doesn’t function) takes a LONG time…then the following:
curl: (28) Failed to connect to 192.168.0.25 port 5055: Connection timed out

Logging into another container on the shared network, I receive the following:
root@b94855dcaf0c:/# curl http://192.168.0.25:5055 -v

  • Trying 192.168.0.25:5055...
  • connect to 192.168.0.25 port 5055 failed: Connection timed out
  • Failed to connect to 192.168.0.25 port 5055 after 129868 ms: Connection timed out
  • Closing connection 0
    curl: (28) Failed to connect to 192.168.0.25 port 5055 after 129868 ms: Connection timed out
    root@b94855dcaf0c:/#

Checking the network address Portainer has assigned to the containers…the one I have mapped to overseerr (port 5055) connects with the following (from within a container on the same Docker network (assigned/created within the stack)
root@b94855dcaf0c:/# curl http://172.18.0.9:5055 -v

  • Trying 172.18.0.9:5055...
  • Connected to 172.18.0.9 (172.18.0.9) port 5055 (#0)

GET / HTTP/1.1
Host: 172.18.0.9:5055
User-Agent: curl/7.81.0
Accept: /

  • Mark bundle as not supporting multiuse
    < HTTP/1.1 307 Temporary Redirect
    < X-Powered-By: Express
    < Location: /login
    < Date: Tue, 19 Dec 2023 00:04:22 GMT
    < Connection: keep-alive
    < Keep-Alive: timeout=5
    < Transfer-Encoding: chunked
    <
  • Connection #0 to host 172.18.0.9 left intact
    root@b94855dcaf0c:/#

And I receive the following, with same IP, from ssh onto the NAS directly:

ter_curl http://172.18.0.9:5055 -v

GET / HTTP/1.1
Host: 172.18.0.9:5055
User-Agent: curl/7.76.0
Accept: /

< HTTP/1.1 307 Temporary Redirect
< X-Powered-By: Express
< Location: /login
< Date: Tue, 19 Dec 2023 00:05:59 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< Transfer-Encoding: chunked
<

REALLY appreciate any other paths I can take / look for…

From the NAS, I also see the below for a netstat -a

netstat -a

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:5055 0.0.0.0:* LISTEN

This is the issue. I’m not sure why it would have stopped working but it could be something like the docker networking changed on the NAS.

Update: Solved!

Nothing that I can recall had changed…aside from me plugging in / adding two additional drives and prepping them in the NAS…however…something in the NAS must have changed…

I did computer guy 101…and rebooted the NAS…and now everything seems to be back working! laugh

Thanks for the sanity check!!

1 Like