Unauthorized: Record for tunnel not found - removed?

Today we experienced an URGENT issue where almost all of our servers were unable to connect to Cloudflare tunnels.

Upon investigation we received this in our logs:

“Unauthorized: Record for tunnel not found”
Unauthorized: Record for tunnel not found",“time”:“2022-02-22T11:32:26Z”,“message”:“Register tunnel error from server side”}

We are able to get it resolved by deleting, then recreating each tunnel however this caused a major outage on all our servers which requires manually reset 100’s of servers by hand.

Any ideas why our tunnels removed? Is there a known issue with Cloudflare tunnels? Did they expire? Anyone else having issues?

My name is Abe and I’m the Product Manager for Cloudflare Tunnel. First, thanks for flagging this. We have not observed any known issues with Cloudflare Tunnel nor should they have any associated expiration period. Would you mind opening a support ticket through https://support.cloudflare.com/hc/en-us/requests so we can ask for more information and throughly investigate. Alternatively, you can email support[at]cloudflare.com and send the ticket number here. We’ll take a look.

#2381682

Thank you. We’re looking into your ticket.

We looked at multiple server and our backend logs and don’t see any evidence we removed sites on our side either manually or programmically. If you need more examples we have other sites we can reference.

So today, one of our machines we purposely left broken is now able to reconnect the tunnel after restarting the Cloudflare Tunnel agent whereas yesterday restarting the Cloudflare Tunnel agent did not resolve the issue. Where there any changes on the Cloudflare side? Were Cloudflare tunnels that got mistakenly removed possibly restored or was some process down yesterday? Either way we should not have to restart the Cloudflare tunnel agent to resolve this issue - it should self resolve.

Negative, nothing known to us.

If you can provide as much detail as possible (cloudflared logs ideally) in the support ticket, hopefully that’ll be forwarded to us and we can analyze.

I posted the log from the machine where I restarted that tunnel agent at {“level”:“info”,“time”:“2022-02-22T13:53:37Z”,“message”:“Tunnel server stopped”}

In the log you will see it recovered and was able to reconnect the tunnel after restarting the service - prior to that it just threw "{“level”:“error”,“connIndex”:2,“error”:“Unauthorized: Record for tunnel not found”,“time”:“2022-02-22T13:53:30Z”,“message”:“Register tunnel error from server side”}

Like the original other server log I posted something happened yesterday morning that is could not get to the edit and got deregistered: Times are GMT (3:29 AM PST).

{“level”:“info”,“connIndex”:0,“time”:“2022-02-22T11:29:36Z”,“message”:“Retrying connection in up to 1s seconds”}
{“level”:“warn”,“connIndex”:0,“error”:“Unauthorized: Record for tunnel not found”,“time”:“2022-02-22T11:29:37Z”,“message”:“Register tunnel error from server side”}

@dgordon1 I think it is best to follow up in the support ticket

However, I have seen the cloudflared log that you shared, and I want to call out that after the cloudflared restart you were using a different tunnel:

cat cloudflared\ \(3\).log | grep 'Starting tunnel'
{"level":"info","tunnelID":"dd291cb5-0db3-4de2-a73a-a63420143516","time":"2022-02-20T08:48:10Z","message":"Starting tunnel"}
{"level":"info","tunnelID":"e3aef07a-17ce-4972-9113-118a4abe40a9","time":"2022-02-22T16:53:11Z","message":"Starting tunnel"}