[SOLVED] Tunnel + SSH + Docker rootless?

Hi everyone,

I migrated to Cloudflare tunnels for my home server recently, and all has been going well. As another refactor, I’ve recently migrated to Docker in rootless (userspace) mode. That’s also gone well, but I now realised I’ve lost my SSH service through the Cloudflare tunnel.

At first I thought this must be because of the standard port 22; the tunnel docker container is now running as a standard user #1000, rather than root as before. So, I adjusted my config to point to port 2222:

   - hostname: ssh.mydomain.net
-    service: ssh://172.22.0.1:22
+    service: ssh://172.22.0.1:2222

I then ran another SSHD instance to test:

/usr/sbin/sshd -p 2222 -ddd

Then tried ssh [email protected], but get the following errors from the Cloudflare container:

2022-05-03T19:03:01Z DBG CF-RAY: 705b41181fb3774d-LHR GET / HTTP/2.0
2022-05-03T19:03:01Z DBG Inbound request CF-RAY=705b41181fb3774d-LHR Header="map[Accept-Encoding:[gzip] Cdn-Loop:[cloudflare] Cf-Access-Authenticated-User-Email:[[email protected]] Cf-Access-Jwt-Assertion:[eyJhbGciOiJSUzI1NiIsImtpZCI6IjMyZjgxNzcyZmVkMTkwMTA2OTQyMWE0MGIwNmNlOTI5ODBjNTMwNTVmMjQ5MTU1ZmUyYjA0YzQ4NzI3ZGYzMjQifQ.eyJhdWQiOlsiMWEzZDU3MmEzZDRjYTRhYjc2NmVhMTA3N2ZjZTdkMDlhMDFlNGZmZDU1N2E5MzczY2Q4OGI1Nzk1NmE2NTQ2MSJdLCJlbWFpbCI6ImpvbmNyb29rZUBnbWFpbC5jb20iLCJleHAiOjE2NTE2NjYzNTAsImlhdCI6MTY1MTU3OTk1MCwibmJmIjoxNjUxNTc5OTUwLCJpc3MiOiJodHRwczovL2pjcm9va2UuY2xvdWRmbGFyZWFjY2Vzcy5jb20iLCJ0eXBlIjoiYXBwIiwiaWRlbnRpdHlfbm9uY2UiOiJCeFJZbjdFWWhhanBxdUY4Iiwic3ViIjoiZDAwZmViYjAtYTQyZS00ZTM5LThhNTMtMjg4NzJhMzRiYWJmIiwiY291bnRyeSI6IkdCIn0.feH8fjci5CGr_rjdvM9etfWZbs1KYWKv-lpW0-_0A3bXXnD4qd3Zp2mjRabp2Q4bK_OLETSXmvC7HSGT0u7X0noxPFX-5h-CMrZ9prDrd_rpflQuxKeG7dhZ8JIodSCR3ZdRSDCcqeBZMdl-iDPMwmab-1-yc8HqV6VYkp1NhckXQFl6J0DG7R6EZx-Q0CYykUbzvjdeGgDOYq7VqcFMOgk9YLNrHiW5x_2XdjZXnph43WVJbICFKmHN7MJisoAU9D3pqRSrzpEZOTb0YmrNDClatiwaNbfbMCICayTcx2J3d-uc-sl_jnDpCJ8ZfnGILQcoFP98FfkMU_JU0Z3lNQ] Cf-Access-Token:[eyJhbGciOiJSUzI1NiIsImtpZCI6IjMyZjgxNzcyZmVkMTkwMTA2OTQyMWE0MGIwNmNlOTI5ODBjNTMwNTVmMjQ5MTU1ZmUyYjA0YzQ4NzI3ZGYzMjQifQ.eyJhdWQiOlsiMWEzZDU3MmEzZDRjYTRhYjc2NmVhMTA3N2ZjZTdkMDlhMDFlNGZmZDU1N2E5MzczY2Q4OGI1Nzk1NmE2NTQ2MSJdLCJlbWFpbCI6ImpvbmNyb29rZUBnbWFpbC5jb20iLCJleHAiOjE2NTE2NjYzNTAsImlhdCI6MTY1MTU3OTk1MCwibmJmIjoxNjUxNTc5OTUwLCJpc3MiOiJodHRwczovL2pjcm9va2UuY2xvdWRmbGFyZWFjY2Vzcy5jb20iLCJ0eXBlIjoiYXBwIiwiaWRlbnRpdHlfbm9uY2UiOiJCeFJZbjdFWWhhanBxdUY4Iiwic3ViIjoiZDAwZmViYjAtYTQyZS00ZTM5LThhNTMtMjg4NzJhMzRiYWJmIiwiY291bnRyeSI6IkdCIn0.feH8fjci5CGr_rjdvM9etfWZbs1KYWKv-lpW0-_0A3bXXnD4qd3Zp2mjRabp2Q4bK_OLETSXmvC7HSGT0u7X0noxPFX-5h-CMrZ9prDrd_rpflQuxKeG7dhZ8JIodSCR3ZdRSDCcqeBZMdl-iDPMwmab-1-yc8HqV6VYkp1NhckXQFl6J0DG7R6EZx-Q0CYykUbzvjdeGgDOYq7VqcFMOgk9YLNrHiW5x_2XdjZXnph43WVJbICFKmHN7MJisoAU9D3pqRSrzpEZOTb0YmrNDClatiwaNbfbMCICayTcx2J3d-uc-sl_jnDpCJ8ZfnGILQcoFP98FfkMU_JU0Z3lNQ] Cf-Connecting-Ip:[83.229.20.216] Cf-Ipcountry:[GB] Cf-Ray:[705b41181fb3774d-LHR] Cf-Visitor:[{\"scheme\":\"https\"}] Cf-Warp-Tag-Id:[a2dcc8df-acc6-4643-86c8-38ab688c7ae5] Sec-Websocket-Key:[BbZPev26OQwZV6tVn1Y4Rw==] Sec-Websocket-Version:[13] User-Agent:[Go-http-client/1.1] X-Forwarded-For:[83.229.20.216] X-Forwarded-Proto:[https]]" host=ssh.jcrooke.net path=/ rule=2
2022-05-03T19:03:01Z DBG CF-RAY: 705b41181fb3774d-LHR Request Content length unknown
2022-05-03T19:03:01Z ERR  error="dial tcp 172.22.0.1:2222: connect: connection refused" cfRay=705b41181fb3774d-LHR ingressRule=2 originService=172.22.0.1:2222

My mac machine should still be using the .ssh/config:

Host ssh.mydomain.net
  ProxyCommand /opt/homebrew/bin/cloudflared access ssh --hostname %h

Does anyone have any idea why it wouldn’t be working? Or what I could try?

Thanks!

J

How are you running Cloudflared, with docker?

Hi. Yes. Specifically this image: Docker Hub. The official one didn’t appear to be able to do networking between containers.

In any case it should be:

Internet → Cloudflare (ssh.mydomain.net) → cloudflared container → ssh port on docker host machine via shared network gateway

Previously, with Docker running as root, this all worked fine using port 22.

Now the only thing that has changed it that Docker is no longer running as root on the host. So, it’s not privileged. I’ve changed the Cloudflare config to point to port 2222, and ran a test instance of sshd to try it out. Then just go the errors above. I don’t really see why anything else should be different…

That is odd, I am able to do networking between containers just fine.

Looking over the limitations of rootless mode and these two stick out

  • IPAddress shown in docker inspect and is namespaced inside RootlessKit’s network namespace. This means the IP address is not reachable from the host without nsenter -ing into the network namespace.
  • Host network ( docker run --net=host ) is also namespaced inside RootlessKit.

It seems like there might be issues with going to the host network from a rootless docker container.

Hey @Cyb3r-Jak3. Good spot. I thought I had other services doing similar things, but good point that I might not to be able to contact the host… I’ll look into that, it does make sense. Thanks, I’ll report back

1 Like

Ok, solved! Yes indeed; I can no longer use the network’s gateway to contact the host machine. Instead I have to use it’s actual IP address. When doing it that way, all works as expected

Thanks for your help @Cyb3r-Jak3

2 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.