Server down due to Tunnel issue

Hi, we have two servers that we are using for load balancing. Both servers are mirrors of each other. Both servers are using:
/usr/bin/cloudflared tunnel --config /tunnels/site1.yml & /usr/local/bin/cloudflared tunnel --config /tunnels/site2.yml & /usr/local/bin/cloudflared tunnel --config /tunnels/site3.yml

Server one is working great.

Server two now shows the error after starting cloudflared:
ERR Initiating shutdown error=“adding 1 origins to current total (4) would surpass the limit (4): validation failed”
INF Metrics server stopped
adding 1 origins to current total (4) would surpass the limit (4): validation failed
INF Connection established connIndex=0 location=IAD
WRN Register tunnel error from server side error=“adding 1 origins to current total (4) would surpass the limit (4): validation failed” connIndex=0
INF Retrying connection in up to 32s seconds connIndex=0
INF Tunnel server stopped

Server two has (cloudflared version 2022.3.4 (built 2022-03-25-1730 UTC))

We did not have this error 1.5 months ago. Both servers would successfully run cloudflared. Did something change with the origin limits or is this another issue. If so can this be adjusted?

Based on the error and the commands, it looks to me that you are running Legacy Tunnels (and so you must have a message in the logs telling you that they are deprecated and will stop work some day).

We encourage you to use Named Tunnels and we’ll be happy to help if you find problems with them: https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/do-more-with-tunnels/migrate-legacy-tunnels

All our docs pages are updated to explain Named Tunnels usage, so if you just follow the docs, you will be using them.

1 Like

Thank you for the information.

We have two servers that we load balance, each has 3 domain names (example.com, sales.example.com, anothersite.com). After both servers have been updated and running the current cloudflared version will this resolve the limit issue? Older configuration worked and we did not have the limit issue.

Also I know each tunnel requires a cert and you have to select a domain name after
cloudflared login. If we select example.com will that same cert work for anothersite.com on the same server? Can we define certs in ingress rules? Was a little confused about this, maybe I over looked how to do this.

Can Legacy Tunnels work along side the new way. We currently have legacy working on a production server and did not want to take their server offline testing the new config rules.

Do you know if limits changed in the last two to 3 months?

Thanks again.

The cert.pem generated by cloudflared tunnel login is your authentication for the account - the tunnel-level authentication is your <TUNNEL-UUID>.json file generated when creating a tunnel.

https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/tunnel-permissions/

The limit regarding 4 connections to Cloudflare’s edge has always been the case as far as I’m aware - at least, I’ve never encountered a scenario where a tunnel has had more than 4.

No.

But it is possible that your Legacy Tunnels got into some bad state (ephemeral or not) due to some bug.
The reality is that we haven’t looked into or supported Legacy Tunnels for a long time, so the way out is to adopt Named Tunnels (which is all the docs refer to https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/).

Thanks!

Thanks @KianNH @nuno.diegues for your help. We upgraded our servers to the latest cloudflared version and used the latest way to create tunnels and map ports. After doing some testing the limit issue was related to the Origin Pools / Origin Servers / Origins. Which matches the error. To add more origin servers we reached out to Cloudflare sales and did not receive a reply. We were able to manually add more by going to “Configure additional features” under Load Balancing. For anyone else that may run into this issue and be a little confused how to resolve.


Screen Shot 2022-04-25 at 12.57.58 PM