Unstable Load Balancer with Argo Tunnel

To reproduce,

  • Make some HTTP application.

  • Create a load balancer.

  • Add origin via an ‘A’ record for HTTP application.

  • Add origin via a ‘CNAME’ record from an Argo Tunnel for HTTP application.

  • Set balance to 50/50. (This value doesn’t really matter)

  • Let it settle until both hosts are ‘Healthy’ .

Now access the HTTP application via the load balancer DNS record, after a couple of minutes start to get intermittent ‘Error 1000’ .

  • Remove A record origin from load balancer. Wait a minute.
  • Refresh HTTP application from load balancer, all works fine.
  • Add A record back in, remove CNAME record.
  • Refresh HTTP application from load balancer, all works fine.

From my testing, it seems if you mix an A record and a CNAME from an argo tunnel in the load balancer, it becomes strangely unstable with the intermittent ‘Error 1000’ . I’ve read the documentation for Error 1000 and I don’t understand how it can be intermittent.

Support ticket ID 2264877 if I have missed anything out.

Thanks

1 Like

@MoreHelp

2264877
Created September 24, 2021 15:34

This sounds like an interesting issue to solve (imo) yet it has had no human activity.

Cheers,

Jon

Even more confusing.

  • Primary LB pool is stable with 2 A records in.
  • Added the Argo tunnelled server into a fallback pool on its own. Thought this would be a safe operation.

Load balancer starts to fail with,

Error 1000
Ray ID: 699efe170c986a35 • 2021-10-06 12:46:26 UTC
DNS points to prohibited IP

Even though the primary pool was healthy, somehow the fallback pool can take down the load balancer.

#confused

17 days on the ticket and no human response - you could at least fob me off for a bit with a ‘we are looking into it’ sort of thing :hugs:

How about a “sorry, this one fell through the cracks.” Tickets tend to get auto-closed. Is yours still open? I’ll put it in the escalation queue just to be sure.

1 Like

Thanks - yes it is still open. Be great if someone can have a look, I get it everyones problems are the end of the world and you need to prioritise, but I’m sure this is a problem your side :stuck_out_tongue:

Cheers

I’m pretty sure that’s not one of the problems I have on my side of the monitor (I don’t work for Cloudflare), so I’m passing the buck. If a support engineer drops in this morning, we’ll be sure to point them in your direction.

2 Likes

Fair enough - I shot the wrong messenger, thanks for any poking of CF staff you can do,

Cheers

1 Like

Might help if you mentioned how you configured that CNAME Argo Tunnel record? Assigned CNAME to your domain or actual tunnelid.cfargotunnel.com Argo Tunnel address. With load balancer, with Argo Tunnel I use CNAME to tunnelid.cfargotunnel.com and override HOST header with domain you want to resolve to i.e. the domain name added to DNS record pointing to tunnelid.cfargotunnel.com. I have CF Loadbalancer with origins from CF Argo Tunnel and CF Worker serverless HTML page as failover backup (i.e. maintenance 503 downtime notice page)

I have,

CNAME
Name: myapiserver (I set this)
Content: tunnel-id-here.cfargotunnel.com

Not sure if that answers your question?

Interesting you say you are using the ‘host override’ in the load balancer with an argo tunnel, I previously had a support ticket open where it was stated that the host override wasn’t working correctly for tunneled origins and instead I had to use the catchall rule of the tunnel config to direct traffic correctly. I didn’t notice this immediately due to a ‘bad’ config on my nginx server and was just luck it was working correctly. Only once I started inspecting the ‘host’ header and enabling verbose logging in cloudflared did I notice this.

edit: @eva2000 - Would you be able to stick an A record in your load balancer, the one with the tunnel on? Be interesting if it starts to fail after a while.

You mean CF A record IP that the Argo tunnel resolves to? That would probably cause DNS points to prohibited IP

Error 1000: DNS points to prohibited IP

Common causes

Cloudflare halted the request for one of the following reasons:

  • An A record within your Cloudflare DNS app points to a Cloudflare IP address, or a Load Balancer Origin points to a proxied record.
  • Your Cloudflare DNS A or CNAME record references another reverse proxy (such as an nginx web server that uses the proxy_pass function) that then proxies the request to Cloudflare a second time.
  • The request X-Forwarded-For header is longer than 100 characters.
  • The request includes two X-Forwarded-For headers.
  • A Server Name Indication (SNI) issue or mismatch at the origin.

Resolution

  • If an A record within your Cloudflare DNS app points to a Cloudflare IP address, update the IP address to your origin web server IP address.
  • There is a reverse-proxy at your origin that sends the request back through the Cloudflare proxy. Instead of using a reverse-proxy, contact your hosting provider or site administrator to configure an HTTP redirect at your origin.

Well seems to work for me been using it for a few months now

image

Like my test setup,

Load Balancer with 2 (or more origins)

One Origin can be the argo tunnel host, another can be another host which is a A record for another server.

Eg I had,
One server which was the only application on the server (and I hadn’t setup a tunnel yet), it had an A record setup, api-server-1 .

Another server which was behind a tunnel, it had a CNAME record of api-server-2 tunnelid.cfargotunnel.com.

api-server-1 and api-server-2 were both added as origins to the load balancer. If they existed together in the LB it was unstable, one or the other though worked fine.

It is bit of an ask really for you to try this, unless you happen to have spare servers around and want to mess with your infrastructure to the point that it possibly breaks, I’m just curious though.

make sure both those if setup as CNAME are not CF orange cloud proxied

Error 1000: DNS points to prohibited IP

Common causes

Cloudflare halted the request for one of the following reasons:

  • An A record within your Cloudflare DNS app points to a Cloudflare IP address, or a Load Balancer Origin points to a proxied record.

ooh that’s interesting, in my case the clouds are orange O_o , but if I turn the lovely orange cloud off, I lose all the CF features to that origin? or I don’t because the load balancer is proxied? it sounds like you are onto something though, will try in a min.

Edit: This is confusing though, because currently I have 2 A records which are proxied and it is working fine

your loadbalancer is proxied so takes care of access that way

you can setup load balancer origins pointing to origin real IP and then use HOST header to set the actual domain name and similarly for argo tunnel set origin to point to tunnelid.cfargotunnel.com and use HOST header to set actual domain name.

2 Likes

Right this is interesting, it seems to be working.

Previously when the tunnel was added in a fallback pool by the CNAME name, proxied, it was taking down the load balancer intermittently. Now that the proxy is disabled for the tunnel, it seems fine.

Both primary origins are proxied and this as no effect on the stability, it has been working like this for months, it is something specific about a proxied tunnel in a load balancer.

Now that the tunnel is not proxied, I can’t access the resource it points to via it’s CNAME record, is this expected?

Thanks for your help

Do it this way

Argo CNAMEs then can remain proxied

Amazing! Working! (doh - I realise what you were saying now)

The main confusion from all of this was the inconsistent results, that it was working some of the time and other times giving the Error 1000, that even in the fallback origin pool the tunnel via CNAME was affecting the primary pools stability. That it would sometimes serve results from a tunnelled proxied origin (via CNAME) just fine, then a few seconds later would fail (Error 1000).

It would be great to know what was going on with CF internally, why it was working/not working.

Thanks very much @eva2000 !

1 Like

Glad to hear it’s working - enjoy Argo Tunnels :smiley: