Cloudflare-Tunnel container cannot talk to other containers in the same docker-network on a NixOS server

I posted this to the Portainer community on Reddit days ago, and the Docker forums the other day and I’m not getting any responses at all. I’m completely stuck and not sure what the do/check next.

I’m super ignorant. With that in mind…

I have Portainer running on a NixOS machine. I have the CloudflareD daemon running in a docker container on that same machine. I configured the non-container-Clourdflare-web-console-side correctly (correct IP address and port number). I can hit the intended domain, the auth-protections I put in place in the Cloudflare web console work, but then I get a 502 bad gateway error when the Cloudflare container attempts to reach the service I want to expose. The service I’m trying to access through the CloudflareD daemon is up and running. It’s available and working correctly when I access it from another machine on the local network that the NixOS machine is connected to.

The issue appears to be that the CloudflareD container cannot talk to the other containers. I setup multiple subdomains on the Clourdflare web console side to confirm that this issue was not just a single container issue. The CloudflareD container cannot talk any other containers.

Both containers, the CloudflareD container and the service I’m trying to expose, are on the “bridge (System)” network. “enable_icc” is true on that network. So they share a network.

I tried to connect to the console of the CloudflareD container via Portainer (using /bin/bash in Containers > cloudflared-tunnel > Console) and nothing happens. When I click the Connect button in that page it flashes “Exec into container as default user using command bash” and then it goes right back to the Containers > cloudflared-tunnel > Console page. No notification, no error message. Checking the logs for that container, there are no new logs generated from that console-connection-attempt.

So I SSH’d to the NixOS machine and tried running the docker commands directly. Running docker container exec -it cloudflared-tunnel bash returns this error: OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown (remember I don’t know anything so…) I believe that means bash isn’t found in the $PATH ENV variable in Containers > cloudflared-tunnel under the “Container details” section. Here is what is listed there: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin .

My hope was to get into the container to see if I could ping the IP address of the container that is running the service I’m trying to expose. Then I learned you can do that with this command: docker exec cloudflared-tunnel ping [the name of the container you are trying to ping] -c2 . Running that command returns this error: OCI runtime exec failed: exec failed: unable to start container process: exec: "ping": executable file not found in $PATH: unknown .

I keep running into road blocks when I try to troubleshoot. I think those road blocks are actually the same issue over and over again (something wrong with the $PATH I guess). I’m just not knowledgeable enough to know what the problem is, where to look for signs of the problem(s), or how to Google-foo my way to an answer, apparently.

Edit: Still unresolved but here’s a touch more info. I thought maybe since its a NixOS machine, docker was looking in a specific place for bash (likely “/bin/bash”). So I ran whereis bash on the machine and it returned /nix/store/(hidden_for_privacy_reasons)-system-path/bin/bash. I was hoping that maybe adding this path to the $PATH would resolve the issue. So I did that via Portainer, restarted the container and still nothing. I also went back to the Containers > cloudflared-tunnel > Console page and entered that /nix/store/ path as a “custom path” for bash and that also didn’t work.

I’m not even sure if thats the issue now. I would think that Nix would create a symlink from “/bin/bash” to that “/nix/store/-system-path/bin/bash” location so that things looking for /bin/bash wouldn’t run into issues. But I could be wrong about that (or literally all of this lol).

You’re going down quite the rabbit hole, troubleshooting. The official cloudflared docker images use distroless images which means stuff like bash, ping etc. doesn’t exist.

The best way to troubleshoot is to do docker logs <cloudflared container name>. You can also run a second container like Debian on the same docker network to troubleshoot connection issues between the container and your host. If you post your tunnel config, I can see if there is anything obvious that would cause issues.

Good morning,

Thanks for the info about the official docker container, thats helpful to know and also makes sense as to why those tools weren’t available.

Portainer happens to display the docker-logs in its container-UI. I did SSH to the server and manually run the docker logs command and it returned the same log that is visible in the Portainer GUI. “Unable to reach the origin service” – which is effectively the same thing as a 502 gateway error.

ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: dial tcp 1xx.1xx.6x.2x:5010: i/o timeout" connIndex=3 dest=https://sb.xxx.xxx/ event=0 ip=1xx.4x.2xx.193 type=http

From your host running cloudflared are you able to load 1xx.1xx.6x.2x:5010 with a tool like curl? Typically that error means the origin service isn’t reachable by cloudflared.

Morning, and thanks for continuing to assist.

From NixOS (the host) itself, I can successfully curl the application. I can also access the application over a browser on other machines that are on my internal network. I know the application/endpoint is up and available as it should be. The CloudflareD-docker-container itself can’t reach any containers (other than itself) at all.

To me, this points to a docker-networking issue (with the caveat that I really barely understand docker at all). Since this is one application running in a docker container trying to talk to another application running in an adjacent docker container on the same host, thats why I think its a docker-networking issue, instead of a physical network issue.

In Googling answers, it appears that a lot of people run into this or similar issue when the applications do not share a network. These two applications are both in 1 network. That network is the default Bridge network that docker creates for itself.

Very good to know.
If both containers are on the bridge network, then they do not get access to use DNS to resolve containers on the same network. If you want a quick and dirty test then run docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' <container which you want cloudflared to connect to> then update the tunnel config to use that IP address. Make sure to use the unmapped port in the tunnel config, ie if you have docker run -p 5010:80 you want to use 80. The caveat is that the IP address of the container can change if it is stopped or recreated.

The better solution, it to make a new docker network and have both containers on it. This way you can use the container name in the tunnel config.

1 Like

Okay so that worked. Thank you!! This makes sense to me now I think. Let me describe this in my own words and you tell me if I’m correctly describing what my problem was.

So because the CloudflareD daemon is running in its own container and trying to talk to another container running on the same host, and same docker-network.

  • If I’m going to configure the tunnel to point at an IP address, the IP address I use in that tunnel configuration has to be an IP that exists inside of that docker-network (the bridge network in this case) that the CloudflareD container is part of.

  • And, regardless of whether I use the IP address of the container or DNS, I need to point the tunnel at the port that the container is exposing to the docker-network.

So my next question for you is, it sounds like by default the default docker bridge network doesn’t have DNS, and that one of the benefits of custom networks is to automatically allow DNS? Is that a fair statement? If so, as far as best practices go, should I create a custom network for things I want to connect to the CloudflareD container?

Yes, but you should avoid doing routing by container IP unless you make the IPs static.

Correct, you want the port that the service is running on, not the port that it is mapped to on the docker host.

Correct.

This is what I have on one of my hosts. I made a separate cloudflared network and which has my cloudflared container and any others I want to use.

1 Like

Hey, Jak3, thanks again. I’m going to link to this post from all the other places I posted this question – none of which have even gotten a single response lol.

I ended up creating a separate cloudflareD network and adding the containers I wanted to expose through cloudflare just before your last reply. And that did fix the DNS issue I was about to run into.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.