Cloudflare secures your origin servers by proxying requests to your hostname through our anycast network and to the external IP of your origin. We launched Argo Tunnel as a secure way to connect your origin to Cloudflare.
Instead of a traditional reverse proxy deployment sending traffic to an external IP, a lightweight daemon runs in your infrastructure and creates outbound-only connections to Cloudflare’s edge. You can disable all ingress and only allow egress.
Your Argo Tunnel connection corresponds to a DNS record in your account. Requests to that hostname still hit Cloudflare’s network first and our edge sends those requests over the Argo Tunnel to your origin. Since these connections are outbound-only, you no longer need to poke holes in your infrastructure’s firewall. Your origins can serve traffic through Cloudflare without any public IP.
However, fitting an outbound-only connection into a reverse proxy added some ergonomic and stability hurdles. Over the last year, the Argo Tunnel team has focused on improving stability and using that as a foundation to make it easier to use. Starting today, we’re releasing a beta of the most significant upgrade to Argo Tunnel’s architecture since launch.
The change starts by avoiding recreating things that should stay persistent. Argo Tunnel now has fewer moving parts when you don’t need them. That change also makes it easier to treat Argo Tunnel like traditional origins. We’re excited to share it with you, but need your help providing feedback.
Keeping persistent objects persistent
Argo Tunnel has objects that tend to stay persistent (DNS records, instances of
cloudflared) and objects that deliberately change and recreate (connections to Cloudflare). Argo Tunnel previously conflated the two, which led to some issues.
The edge vs the control plane
Cloudflare as a whole consists of two components: the edge network and the control plane that manages the configuration of that network.
The data centers in 200 cities around the world that proxy traffic to your origin make up the edge network. These data centers are highly available and, thanks to AnyCast IP routing, can gracefully handle traffic if one or more data centers go offline.
The control plane pushes configuration to the edge. When you make a change to something in Cloudflare (whether via the UI in Cloudflare’s dashboard, or the API) our control plane receives it, authenticates it, and then pushes it to our edge.
If the control plane goes down, the edge should not be degraded - traffic will continue to be served using the most recent configuration. At launch, Argo Tunnel muddled the two in some places, which meant that control plane issues could become edge issues for Tunnel users.
Starting every Tunnel from scratch
Regardless of whether a Tunnel is connecting for the first time or the 100th, the operation repeated a series of high-level steps:
cloudflaredconnects to an Argo Tunnel service running in Cloudflare’s control plane. That service registers your Tunnel and its connections.
- The service creates a public DNS record for your hostname which points to a randomly generated CNAME record for load balanced Tunnels or an IPv6 for traditional Tunnels. The ephemeral CNAME record represents your Tunnel.
- The control plane then tells Cloudflare’s edge about these DNS entries and where the CNAME or IP address should send traffic. Traffic can now be routed to
- If the Tunnel disconnects, for any reason, the Argo Tunnel service unregisters the Tunnel and deletes the DNS record.
The last step is an issue. In most cases, you create an Argo Tunnel for a service meant to run indefinitely. The DNS record should stay persistent. - it’s an app that you manage that should not change However, a simple restart or disconnection meant that
cloudflared had to follow every step and start itself from scratch. If any of those upstream services were degraded, the Tunnel would fail to reconnect.
This model also introduces other shortcomings. You cannot gracefully change the DNS record of a Tunnel. Visibility was limited. Load balancing introduced complications with how origins were counted.
Phase 1: Improving stability
The team started by reducing the impact of those dependencies. Over the last year, Argo Tunnel has quietly replaced single points of failure with distributed systems that are more fault tolerant.
Tunnels now live longer. Argo Tunnel has migrated to Cloudflare’s Unimog platform, which has increased the average life of a connection from minutes to days. When connections live longer, they restart less, and are then subject to fewer upstream hiccups.
Additionally, some Tunnels no longer need to follow the entire creation flow. If your Tunnel reconnects, we opportunistically try to reestablish it with the records already at our edge. However, that succeeds in a minority of cases because it relies on your Tunnel hitting the same parts of our edge network.
These changes have dramatically improved the stability of Argo Tunnel as a platform, but still left a couple of core problems: Tunnel reconnections were treated like new connections and we still had too many upstream dependencies.
Phase 2: A new look for Argo Tunnel
Solving those core problems requires distinguishing between the mostly persistent objects (DNS records,
cloudflared) from the ephemeral objects (the connections themselves). Argo Tunnel’s new architecture does so by introducing the concept of a name.
In the old model, nearly every aspect of the connection to Cloudflare had to be recreated every time.
In this new model, you can create an Argo Tunnel that has a persistent, stable name. You can also create a corresponding persistent DNS record, either from
cloudflared or the UI/API, that points to that name. When connections are disrupted or restart, neither the named Tunnel or DNS record need to be reestablished.
Instead, those connections reach back out to the assigned name and begin proxying traffic again, reducing the upstream dependencies on DNS provisioning and the Argo Tunnel service in our control plane. The breakdown below walks through the difference and what this means for stability in Argo Tunnel.
How it works
You can begin using this new architecture today with the following steps. First, you’ll need to upgrade to the latest version of
1. Login to Cloudflare
cloudflared tunnel login and authenticate to your Cloudflare account. This will give your instance of
cloudflared the ability to create Named Tunnels in your account, as well as the ability to eventually point DNS records to them. If you’ve already run
cloudflared login before, you might need to delete your cert.pem as Named Tunnels uses a new cert format. The new format is backwards-compatible and can also be used to start classic tunnels.
2. Create your Tunnel
You can now create a Tunnel that has a persistent name. Run
cloudflared tunnel create <name> to do so.
cloudflared will create a Tunnel with the name that you give it and a UUID. This name will be associated with your account and you can connect any enrolled instance of
cloudflared to this name.
3. Associate your Tunnel with a DNS record
You can now associate this persistent Tunnel object with an equally persistent DNS record. Run the following command,
cloudflared tunnel route dns <name> <hostname> or
cloudflared tunnel route dns <UUID> <hostname> where the UUID is the value generated in step 2 and the hostname is a subdomain of a domain in your Cloudflare account.
That subdomain will now point to this UUID/name, regardless of Tunnel restarts or disruptions.
4. Configure Tunnel details
Configure your instance of
cloudflared, including the URL that
cloudflared will proxy traffic to in the configuration file.
5. Send traffic to your Tunnel
Your named Tunnel now has a persistent name and UUID, a DNS record it receives traffic from, and a local destination it sends traffic to… You can tell it to begin running with the command
cloudflared tunnel run <name> or
cloudflared tunnel run <UUID> and it will start proxying traffic.
When this instance of
cloudflared restarts, the name, UUID, and DNS record will not need to be recreated. The connection will reestablish and begin serving traffic.
[Optional] Check what Tunnels exist
You can also use this architecture to see your active Tunnels. Run
cloudflared tunnel list to view the Tunnels created and their connection status. You can delete Tunnels, as well, by running
cloudflare tunnel delete <UUID>.
Over the next week we’ll be releasing an update to allow you to complete steps 1-5 in a single command, including the URL configuration. Additionally, an update will make it possible to replace the commands that reference a UUID with the name you assigned the Tunnel.
Why is this better?
- You can create DNS records that never need to change. You can even create CNAME records via the UI or API and point them to named Tunnels, similar to an IP address.
- When Tunnels reconnect, they do not need to recreate the same DNS records, removing that upstream dependency.
- You now have visibility into your Tunnels and which are serving traffic.
- Once released, you can enroll a Tunnel into a Load Balancer pool as if it were a single origin, simplifying health checks and LB management.
Our new favorite:
You can use
tunnel list to monitor your Named Tunnels; example below:
The new Argo Tunnel architecture is only available for Tunnels that do not use Cloudflare’s load balancer. We will be releasing load balancing support soon as part of the GA release. Please try it out with non-production services and leave us some feedback!