I’ve moved several of our servers behind a firewall and now use SSH to connect to them via Cloudflare tunnel.
Previously, I was making an SSH connection directly to the public IP address of the server and I was usually able to keep connections open for days. But now the cloudflared proxy keeps dropping connections and I’m lucky if it stays up for an hour. This is very annoying because I do most of my work from the command line and now I have to constantly re-login.
Constantly re-establishing connections may not be a problem for HTTP, but it’s a major pain when you’re trying to get something done with a command line SSH.
I cannot find any documentation on the “access” mode of cloudflared. It is possible to make it create multiple connections when it is used as a proxy (like it does in “tunnel” mode)? Then it could keep my SSH terminal alive if one of the connections drops.
Or is there another way to fix this problem? I’ve already set SSH keepalive options, etc and it does not seem to have any effect.
I know this isn’t the answer to your question. I didn’t like the SSH approach that cloudflared offers so I don’t use that but instead I route the IP blocks of the servers I manage and I use ssh directly to the routed IPs. But besides that, what I think you should use is mosh. Once you use mosh there’s no back to regular ssh.
What do you mean by “I route the IP blocks of the servers I manage and use ssh directly to the routed IPs”? My servers are behind a firewall and only have private IP addresses. How can I route to IPs behind the firewall?
My server is behind a NAT gateway and only has one private IP (10.0.0.21). I added this to my cloudflared tunnel as a private network (although it does not seem necessary, since I only have one IP address). I’m managing my Cloudflare tunnel from the web dashboard. When I upgraded, I removed the old manually-installed configuration stuff. The tunnel works fine and I can access all the websites the server handles (on Cloudflare’s public IP address).
In order to access the private IP address of the server, I installed WARP on my client machine (Mac Laptop) and connected to my team. However, I still cannot connect to the 10.0.0.21 IP address or ping it.
WARP shows it is connected (on both the client and on the CF dashboard), but I cannot see how it is supposed to route to the private IP address. When I check the routes on my laptop with route get 10.0.0.21, it shows the gateway as my local router.
Do I need to create an access policy for SSH on this server? What’s missing?
Once your private IP block is added to cloudflared using cloudflared tunnel route ip add CIDR TunnelName you also need to use Split Tunnel by going to Zero Trust dashboard->Settings->Network->Split Tunnel. If you have it set to Exclude, then make sure to not have the 10.x.x.x network there. If you have it set to Include then make sure it is there.
Note that you are not limited to route IPs that belong to a specific network. For example, if you route 17.0.0.0/8 you’ll be routing all Apple, so when you’re using WARP and you connect to apple.com they’ll see the connection comes from the server running cloudflared.
Hello,
ping won’t work as ICMP isn’t included in the tunnel.
You also need to go to your Zero Trust Dashboard, then to Settings->Network and on the Firewall group enable Proxy. Once you save this it can take a while, from minutes to hours to have it applied, and then it’ll work. You can enable UDP if you need to hit any DNS on the other side of the tunnel.
I enabled the Network Proxy setting. It took effect after a few hours and I was was able to SSH to the private IP 10.0.0.121.
The bad news is that I could not connect to any other website from my laptop (not even Cloudflare). I kept getting ‘“Gateway Intermediate ECC Certificate Authority” certificate is not trusted’ errors.
It appears that when the proxy is enabled, it tries to proxy everything, not just my private IP traffic. It looks like a MITM attack and is blocked.
When I disabled WARP, I was able access everything normally. But I could no longer access my private internal IP address.
So I am I forced to choose between one or the other? Or are there other magic settings I can tweak?
If I were doing this 100% on Linux and setting up my own tunnels, I could use ip route and iptables and customize everything. However, I don’t know exactly what CF WARP and tunnel are doing behind the scenes. It’s mostly a black box.
I believe this is the issue. Cloudflare’s network is pretty dynamic, with hardware reboots at various times, plus a general aversion to leaving a connection open indefinitely.
Is there a reason you need an always-on SSH connection to your server?
That’s not an issue. Idle SSH connections don’t use networking resources and don’t get affected by network equipment being rebooted. You can try this by establishing a SSH connection and then rebooting your network AP, you’ll see once you AP is back your SSH is still up. As long as your IP doesn’t change, all stays up. It does, however, get disconnected if you try pressing keys while the connection is gone.
I like leaving my connections open, it makes my life easier.
What you say is true, but I think CF sometimes changes IP addresses or resets connections. Their network is designed mostly for short-lived HTTP connections, so it usually doesn’t make any difference. However, I do most of my work on the command line using vim, so it’s important to have a connection that does not disappear every few minutes.
I have not looked at the source code for cloudflared but when it runs on a server, it appears to be establishing multiple connections to various POPs. I assume this is done so they can have a backup if one of the connections goes away. However, when cloudflared is run as a client, it only establishes one connection. If that one goes away, it has to reconnect and I lose my SSH connection.
It seems to me that they could establish multiple tunnels when running as a client and handle any disconnections and reconnections in a manner that is transparent to the client. I have a dual-WAN Peplink router that does this and it works almost flawlessly. I can pull the plug on one WAN connection and my SSH connection is not affected. My router actually sends redundant traffic over each WAN and uses various kinds of error-correction to detect any problems. The two connections are routed to another router in a datacenter that is close to my server. This router then makes the connection to the public IP on my server.
This was all working fine when I was connecting to a public IP on my servers. However, when I moved our servers behind a firewall and started connecting via a Cloudflare tunnel, it became the weak link in the chain and my failsafe SSH connection no longer worked.
Cloudflare has all the necessary infrastructure to create the same kind of failsafe connections as my Peplink router, but perhaps no one there sees a need for it at the client level. I suppose I could examine the cloudflared source code and try to figure out a way to do this myself, but I do not know Go and have not hacked low-level TCP/UDP networking stuff in over 20 years. Someone who is more familiar with the code could probably accomplish this much quicker.
I was able to get things to work with the proxy by setting “Split Tunnel” to “Include” instead of “Exclude”.
My “Include” is just the IPs of my private addresses (10.0.0.*) that are behind the firewall. Everything else is routed normally. WARP is not trying to proxy everything. Not sure, maybe I don’t even need proxy now (have not tested this).
Yeah, I tried and discovered the same thing. I also tested to see if the WARP connection is more reliable than the cloudflared SSH proxy. I set up two IPs on my server. One connects via the proxy and the other uses WARP.
SSH connects fine with both, but when the connection drops, both of them seem to disappear around the same time. This proves I can’t rely on CF to maintain a solid connection (which I already knew), so I’ll have to use something like mosh or etserver. I used to use etserver and had problems, but it’s still installed, so maybe I’ll update to the latest version and give it another try.
