Argo Tunnel - "Named Tunnel" beta

Cloudflare secures your origin servers by proxying requests to your hostname through our anycast network and to the external IP of your origin. We launched Argo Tunnel as a secure way to connect your origin to Cloudflare.

Instead of a traditional reverse proxy deployment sending traffic to an external IP, a lightweight daemon runs in your infrastructure and creates outbound-only connections to Cloudflare’s edge. You can disable all ingress and only allow egress.

Your Argo Tunnel connection corresponds to a DNS record in your account. Requests to that hostname still hit Cloudflare’s network first and our edge sends those requests over the Argo Tunnel to your origin. Since these connections are outbound-only, you no longer need to poke holes in your infrastructure’s firewall. Your origins can serve traffic through Cloudflare without any public IP.

However, fitting an outbound-only connection into a reverse proxy added some ergonomic and stability hurdles. Over the last year, the Argo Tunnel team has focused on improving stability and using that as a foundation to make it easier to use. Starting today, we’re releasing a beta of the most significant upgrade to Argo Tunnel’s architecture since launch.

The change starts by avoiding recreating things that should stay persistent. Argo Tunnel now has fewer moving parts when you don’t need them. That change also makes it easier to treat Argo Tunnel like traditional origins. We’re excited to share it with you, but need your help providing feedback.

Keeping persistent objects persistent

Argo Tunnel has objects that tend to stay persistent (DNS records, instances of cloudflared) and objects that deliberately change and recreate (connections to Cloudflare). Argo Tunnel previously conflated the two, which led to some issues.

The edge vs the control plane

Cloudflare as a whole consists of two components: the edge network and the control plane that manages the configuration of that network.

The data centers in 200 cities around the world that proxy traffic to your origin make up the edge network. These data centers are highly available and, thanks to AnyCast IP routing, can gracefully handle traffic if one or more data centers go offline.

The control plane pushes configuration to the edge. When you make a change to something in Cloudflare (whether via the UI in Cloudflare’s dashboard, or the API) our control plane receives it, authenticates it, and then pushes it to our edge.

If the control plane goes down, the edge should not be degraded - traffic will continue to be served using the most recent configuration. At launch, Argo Tunnel muddled the two in some places, which meant that control plane issues could become edge issues for Tunnel users.

Starting every Tunnel from scratch

Regardless of whether a Tunnel is connecting for the first time or the 100th, the operation repeated a series of high-level steps:

  1. cloudflared connects to an Argo Tunnel service running in Cloudflare’s control plane. That service registers your Tunnel and its connections.
  2. The service creates a public DNS record for your hostname which points to a randomly generated CNAME record for load balanced Tunnels or an IPv6 for traditional Tunnels. The ephemeral CNAME record represents your Tunnel.
  3. The control plane then tells Cloudflare’s edge about these DNS entries and where the CNAME or IP address should send traffic. Traffic can now be routed to cloudflared.
  4. If the Tunnel disconnects, for any reason, the Argo Tunnel service unregisters the Tunnel and deletes the DNS record.

The last step is an issue. In most cases, you create an Argo Tunnel for a service meant to run indefinitely. The DNS record should stay persistent. - it’s an app that you manage that should not change However, a simple restart or disconnection meant that cloudflared had to follow every step and start itself from scratch. If any of those upstream services were degraded, the Tunnel would fail to reconnect.

This model also introduces other shortcomings. You cannot gracefully change the DNS record of a Tunnel. Visibility was limited. Load balancing introduced complications with how origins were counted.

Phase 1: Improving stability

The team started by reducing the impact of those dependencies. Over the last year, Argo Tunnel has quietly replaced single points of failure with distributed systems that are more fault tolerant.

Tunnels now live longer. Argo Tunnel has migrated to Cloudflare’s Unimog platform, which has increased the average life of a connection from minutes to days. When connections live longer, they restart less, and are then subject to fewer upstream hiccups.

Additionally, some Tunnels no longer need to follow the entire creation flow. If your Tunnel reconnects, we opportunistically try to reestablish it with the records already at our edge. However, that succeeds in a minority of cases because it relies on your Tunnel hitting the same parts of our edge network.

These changes have dramatically improved the stability of Argo Tunnel as a platform, but still left a couple of core problems: Tunnel reconnections were treated like new connections and we still had too many upstream dependencies.

Phase 2: A new look for Argo Tunnel

Solving those core problems requires distinguishing between the mostly persistent objects (DNS records, cloudflared) from the ephemeral objects (the connections themselves). Argo Tunnel’s new architecture does so by introducing the concept of a name.

In the old model, nearly every aspect of the connection to Cloudflare had to be recreated every time.

In this new model, you can create an Argo Tunnel that has a persistent, stable name. You can also create a corresponding persistent DNS record, either from cloudflared or the UI/API, that points to that name. When connections are disrupted or restart, neither the named Tunnel or DNS record need to be reestablished.

Instead, those connections reach back out to the assigned name and begin proxying traffic again, reducing the upstream dependencies on DNS provisioning and the Argo Tunnel service in our control plane. The breakdown below walks through the difference and what this means for stability in Argo Tunnel.

How it works

You can begin using this new architecture today with the following steps. First, you’ll need to upgrade to the latest version of cloudflared.

1. Login to Cloudflare

Run cloudflared tunnel login and authenticate to your Cloudflare account. This will give your instance of cloudflared the ability to create Named Tunnels in your account, as well as the ability to eventually point DNS records to them. If you’ve already run cloudflared login before, you might need to delete your cert.pem as Named Tunnels uses a new cert format. The new format is backwards-compatible and can also be used to start classic tunnels.

2. Create your Tunnel

You can now create a Tunnel that has a persistent name. Run cloudflared tunnel create <name> to do so. cloudflared will create a Tunnel with the name that you give it and a UUID. This name will be associated with your account and you can connect any enrolled instance of cloudflared to this name.

3. Associate your Tunnel with a DNS record

You can now associate this persistent Tunnel object with an equally persistent DNS record. Run the following command, cloudflared tunnel route dns <name> <hostname> or cloudflared tunnel route dns <UUID> <hostname> where the UUID is the value generated in step 2 and the hostname is a subdomain of a domain in your Cloudflare account.

That subdomain will now point to this UUID/name, regardless of Tunnel restarts or disruptions.

4. Configure Tunnel details

Configure your instance of cloudflared, including the URL that cloudflared will proxy traffic to in the configuration file.

5. Send traffic to your Tunnel

Your named Tunnel now has a persistent name and UUID, a DNS record it receives traffic from, and a local destination it sends traffic to… You can tell it to begin running with the command cloudflared tunnel run <name> or cloudflared tunnel run <UUID> and it will start proxying traffic.

When this instance of cloudflared restarts, the name, UUID, and DNS record will not need to be recreated. The connection will reestablish and begin serving traffic.

[Optional] Check what Tunnels exist

You can also use this architecture to see your active Tunnels. Run cloudflared tunnel list to view the Tunnels created and their connection status. You can delete Tunnels, as well, by running cloudflare tunnel delete <UUID>.

Over the next week we’ll be releasing an update to allow you to complete steps 1-5 in a single command, including the URL configuration. Additionally, an update will make it possible to replace the commands that reference a UUID with the name you assigned the Tunnel.

Why is this better?

  • You can create DNS records that never need to change. You can even create CNAME records via the UI or API and point them to named Tunnels, similar to an IP address.
  • When Tunnels reconnect, they do not need to recreate the same DNS records, removing that upstream dependency.
  • You now have visibility into your Tunnels and which are serving traffic.
  • Once released, you can enroll a Tunnel into a Load Balancer pool as if it were a single origin, simplifying health checks and LB management.

Our new favorite: tunnel list

You can use tunnel list to monitor your Named Tunnels; example below:

Screen Shot 2020-09-01 at 7.31.32 PM

What’s next?

The new Argo Tunnel architecture is only available for Tunnels that do not use Cloudflare’s load balancer. We will be releasing load balancing support soon as part of the GA release. Please try it out with non-production services and leave us some feedback!

9 Likes

Will definitely take a look. Love this! I have so many Tunnels active with the various long hostnames that are a pain to manage.

How could I run the same UUID when it is run in service mode?

Hey @dnwk - do you mind elaborating? What do you mean the same UUID in service mode?

This is what I am referring to https://developers.cloudflare.com/argo-tunnel/reference/service/
Running the cloudflared as a systemd service. I can’t specify the UUID in the configuration file.

Got it.

The Tunnel Name, the UUID, is an argument and not an option, so it cannot be specified in the config file.

Instead, you can specify all options about the tunnel in the config file and then use
cloudflared tunnel run <uuid> with the config file specified and it will pick them up.

1 Like

Looking forward to trying this with Cloudflare’s Load Balancers. When might this be GA?

I’m wondering if these improvements make it reliable enough to use for Kubernetes nginx ingress, (with cloudflared as a sidecar) rather than spinning up a Kubernetes external load balancer.

This may already be possible, but it would be great if we could wildcard a subdomain into the Cloudflare load balancer, directing traffic over argo tunnel into the cluster ingress.

We’ll be launching it with LB support in the next few weeks; late Sept or early October.

Thanks for the feedback on the wildcard. That makes sense. Will see what we could do about it.

3 Likes

Have not had time to test yet, taken a quick look only. What would the .service file look like? Basically normal config file, except for the run command itself mentioning the UUID?

Basically instead of cloudflared tunnel --config <path> you put cloudflared tunnel run <UUID> --config <path>? No way to put it in the config, but is it even planned or no? It makes making changes easier.

Correct; it would look like cloudflare’s tunnel —config <path> run <UUID>. Makes sense why adding the config would be easier; not planned yet, but we’ll take a look.

1 Like

Nice article.
You donot have to make any changes inbound in ur firewall to allow the tunnel traffic as the connection is initiated from the Argo VM, internal to ur network.
Where should this VM be deployed? outside DMZ, Same VLAN as the servers? what is the best practice from security perspective?
If the connection is initiated from inside and its encrypted - it means that all my perimeter controls will not apply to this connection e.g multi-layer Firewalls, IPS, etc? Do we have an argument here that even in on-prem solution SSL traffic is not inspected by Firewall - SSL connection is terminated on LB/WAF box and than to webserver? We are in Zero Trust erra :slight_smile:
What is the best practice?

That is a question for the security team of the specific datacenter, each one is configured differently and can have a positive or negative answer to all the questions there, depending on the configuration.

I just want to point out we have setup an example to use named tunnel as a sidecar https://github.com/cloudflare/argo-tunnel-examples/tree/master/named-tunnel-k8s.

1 Like