Argo Tunnel Authorization Improvements

I have some questions and improvement requests regarding the Cloudflare Tunnel authorization process. Right now, as I understood it, the only way to authorize a new tunnel is to run “Cloudflared login”, open the given link in a browser that is logged in to the proper Cloudflared account, and authorize the tunnel. Following this authorization, a certificate is generated that will then be used by the tunnel.

I’ve had issues where the certificate doesn’t seem to get generated, so the Cloudflared executable gets stuck waiting for authorization, even if I authorized it in the Cloudflare account using the link. I tried multiple times today on a Windows machine, until I started getting “too many requests” errors in the Cloudflare authorization page.

Aside from the fact that it does not seem to be working all the time, would there be a way to manually generate a key pair that would be bound to a single subdomain, for the sake of copying the key pair to the machine that will run the tunnel? The idea would be to generate one key pair per tunnel (myapp1.mydomain.com) and make sure that this particular key pair can only be used for the “myapp1” subdomain and nothing else. Ideally, we should be able to track valid key pairs that have been generated and revoke access. This way, the authorization process could be made separate from logging in to the Cloudflare account every time (key pairs could be generated in advance).

In the current Cloudflare Tunnel feature, it looks like one can request any subdomain once authorized. I don’t see an easy way to manage authorizations in the current Cloudflare page. Is there a way to see the current status of argo tunnels, like the current active tunnels, the authorized key pairs, etc?

Thanks!

4 Likes

I am digging this post, but part of the requested enhancements are still not there.

We now have the possibility to generate Cloudflare Tunnel certificates though https://dash.cloudflare.com/argotunnel, great!

However, we are still lacking the possibilities to:

  • track - per host would be great, but also for example to monitor certificate expiration, even if expiration dates are set in the far future
  • revoke generated certificates - this one is really critical, as a leaked X.509 Client Certificate would allow anyone having it to write to the associated DNS zone, especially as they have a really long validity (15 years!)

Any plan on those capabilities?

Thanks!

Aren’t they listed in the Origin Certificates in the SSL/TLS app with expiration dates? They should also be regenerated automatically. They have always been there if I recall correctly. I can see there is an issue in that you can’t distinguish them between various tunnels.

Aren’t they listed in the Origin Certificates in the SSL/TLS app with expiration dates?

Thanks @matteo for pointing them out. Either I overlooked something in the Access / Argo documentation, or it simply isn’t mentioned there. However, this is insufficient.

Cloudflare Tunnel certificates have more info than what you can found in the SSL/TLS app, especially an Cloudflare Tunnel token, as their structure is as follow:
----BEGIN PRIVATE KEY-----
[…]
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
[…]
-----END CERTIFICATE-----
-----BEGIN Cloudflare Tunnel TOKEN-----
[…]
-----END Cloudflare Tunnel TOKEN-----

They should also be regenerated automatically. They have always been there if I recall correctly.

In that case, i.e. PKI, should it be accurate, I consider the automated unrequested regeneration as an issue rather than a solution. Also, the fact, should it be accurate, that a new certificate would be generated automatically to replace the expired one doesn’t push the new one to the orign(s) where it should replace the old one.

I can see there is an issue in that you can’t distinguish them between various tunnels.

Indeed, it is not possible to distinguish, because certificates generated with Cloudflared login or https://dash.cloudflare.com/argotunnel are wildcard certificates.
That is one issue.

Then there is the issue with the Cloudflare Tunnel Token, part of the certificate (cf. PEM structure above).

Let’s take an example when you have 2 Cloudflare Tunnel certificates, A and B, created either through Cloudflared login or https://dash.cloudflare.com/argotunnel, both appearing in the SSL/TLS app:

  • the token A from certificate A can be reused in certificate B, replacing token B
  • after revocation of certificate A in the SSL/TLS app, the token A still works when used in certificate B

In short: back to the initial point, how do we revoke generated certificates, tokens included?

For Cloudflare staff: a minor link correction to do in the SSL/TLS app of the dashboard, regarding Origin Certificates API:

I believe, but obviously never tried since it’s never happened to me due to time, that they would be re-requested by the Cloudflared executable which would pull them down to substitute old one.

As far as revocation I don’t know if they are actually revoked once you de-register a tunnel, but I believe that is not the case.

One way, even though it doesn’t solve the issue completely, since you can’t distinguish between tunnels, would be to actually do custom ones for your origins including the exact subdomains you need and not doing them as wildcards.

I don’t know what those tokens are, I can’t comment on that. It could simply be a random ID completely unrelated to the certificate. No clue on that.

I don’t know what those tokens are, I can’t comment on that. It could simply be a random ID completely unrelated to the certificate. No clue on that.

No worries @matteo, and thanks for trying!

From what I experimented with: the Cloudflare Tunnel Token is required to establish an Cloudflare Tunnel network link with Cloudflared tunnel; if the token is missing from the certificate, the tunnel will not be established, resulting in an error="Server error: Missing token in the certificate".
So, as the regular Origin Certiciates, for which it is indeed possible to specify an FQDN, are lacking this token, they are not usable per se for Cloudflare Tunnel use cases.

As experienced and shared in my previous message, token A from certificate A can be reused in certificate B, so we can imagine, as suggested, to generate an Origin certificate with a precise FQDN, and then put a valid token into it, maybe it could work.
But, besides this manipulation being not appropriate for production with dozens/hundreds/thousands of Argo Tunnels, the fact we can reuse the token such way / not revoke it is an issue.

1 Like

That I can definitely see, there should be a way to distinguish them.

Maybe @cloonan can ping the right person internally.

1 Like

Thanks for your inputs guys, I think it is very important to have the ability to manage Tunnel access per subdomain, as it is often the case that multiple independent services (often belonging to different security zones or groups) are deployed in the same DNS zone. They may even be operated by multiple sub-teams or even organizations. Following the security principle of least privilege it should not be made so easy for an individual application to cause wider damage affecting all services in the DNS zone, if compromised. Currently, once the Tunnel is authorized, it can even change the subdomain it is bound to anytime.

The impact is fortunately limited to:

  • subdomains that don’t exist yet (not critical)
  • subdomains already configured with (other) Argo Tunnels (with or without LB)

If a subdomain does already have an A or CNAME record defined in the DNS section, then an attempt to seize it via such hacked server with valid Argo Tunnel credentials won’t be successful and will end up with the following error:

{"connectionID":"0","error":"Server error: cannot create CNAME: IAPI returns 400 Error for POST https://iapi.cfops.it/v4/zones/xxxx/dns_records: {\"success\":false,\"errors\":[{\"code\":81053,\"message\":\"An A, AAAA or CNAME record already exists with that host.\"}],\"messages\":[],\"result\":null}","level":"error","msg":"Register tunnel error from server side","time":"2019-07-27T18:46:33+02:00"}

However I believe this is more of a lucky coincidence than a proper security measure.
Furthermore, it is not effective if the victim subdomain is also served through Argo Tunnel .
At first it may seem that a service served through a single Argo Tunnel (without LB) is immune to this attack as well, because an attempt to take over the service from the hacked server ends up with the following error:

{"error":"Server error: There is already an active tunnel for victim.example.com. To distribute requests between multiple origins, use Cloudflare Load Balancer. Existing tunnels not using the Load Balancer will need to be registered again","level":"error","msg":"Quitting due to error","time":"2019-07-27T18:36:47+02:00"}

This can however be easily overcome if an attacker declares to use a load balancer in his malicious Argo Tunnel config:

hostname: victim.example.com
lb-pool: seized-pool
hello-world: true

The LB will take precedence (tested, works) and all the requests will be routed to the hacked server only.
Finally, if the victim service is already deployed on multiple servers with Argo Tunnels and a Load Balancer enabled, the attacker can join the pool and have at least a portion of the requests routed to him.

We’re trying to deploy Argo at scale, and have run into the issue that it is not possible to provision Argo tokens using the API (nor terraform).

Furthermore, security concerns outlined by @jknaperek are very concerning. It is critically important that argo tokens be scoped to the most granular permissions/subdomains.

It seems using the undocumented api you can generate your own certificate that is scoped to a particular set of hostnames. See how we’ve implemented it in an ansible role: