Cloudflare Argo Tunnel misbehaving over LTE

Hi,

This is less of a question and more looking for a discussion, at my workplace we frequently use Argo tunnels to connect infrastructure, and we were recently setting up a Raspberry Pi with an LTE dongle for deployment at one of our client sites. We had configured the tunnel as follows (Hostnames and other details have been replaced):

tunnel: xxxx
uuid: xxxxxx-xxxx-xxxxx-xxxx-xxxxxxxxxx
credentials-file: /xxxx/xxxx/xxxx.json
logfile: /var/log/cloudflared.log
loglevel: debug
autoupdate-freq: 24h

ingress:
  - hostname: "ssh.example.xyz"
    service: "ssh://localhost:22"

  - hostname: "switch-ssh.example.xyz"
    service: "ssh://10.0.0.1:22"

  - hostname: "console.example.xyz"
    service: "tcp://localhost:7001"

  - hostname: "caddy.example.xyz"
    service: "http://localhost:80"

  - hostname: "prometheus.example.xyz"
    service: "http://localhost:9090"

  - hostname: "alertmanager.example.xyz"
    service: "http://localhost:9093"

  - hostname: "juniper-metrics-exporter.example.xyz"
    service: "http://localhost:9014"

  #catch all rule
  - service: "http_status:404"

Which worked fine, all the services load when we were connected to the office network via Ethernet. However we noticed when we switched to using the LTE dongle, all HTTP services except the juniper-metrics-exporter.example.xyz failed to load. None of the other HTTP services loaded, and would give 5xx errors, or show no content or error message at all, just a blank web page. We know there’s no issue with any of the applications, as all HTTP services work perfectly when connected via Ethernet.

We solved this issue by changing the tunnel protocol to quic:

tunnel: xxxx
uuid: xxxxxx-xxxx-xxxxx-xxxx-xxxxxxxxxx
credentials-file: /xxxx/xxxx/xxxx.json
logfile: /var/log/cloudflared.log
loglevel: debug
autoupdate-freq: 24h
protocol: quic

However the question remains as to why. None of the HTTP services in question are serving UDP-specific content, they’re all just fairly plain HTML/JS web applications. They should work fine over the http2 protocol which the tunnel uses by default, as far as we’re aware.

Does anyone have any insight they could provide? I don’t mind sharing more details if I can.

QUIC runs over UDP. It seems unlikely, but HTTP/2 runs over TCP - does your provider have any restrictions or filtering on TCP connections?

The provider is Telus, it’s technically possible but unlikely based on their listed blocked ports. I suppose the only way to know for sure would be to contact Telus directly, but I don’t think we would get a timely response (based on experience).

It just seemed fishy that the :9014 service was working over LTE, but the :9090 and :9093 services were not. Logically I would assume if they were blocking ports in this range, it would be blocking something to the effect of 9xxx, not simply a few ports in the low 9000s (again, technically possible, just sounds unlikely).

Since the only thing changing here is your connectivity, that does point to an issue with that connectivity provider rather than an issue with Cloudflare Tunnel - though it could be a path issue between the provider and Cloudflare - but we establish resilient connections to 2 different locations for this reason. You could turn the loglevel up and see what specifically is happening from the tunnel point of view:

https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/arguments/#loglevel

But I think speaking to your provider is probably the best course of action here.

1 Like

That’s totally fair, it does sound like an issue between the provider and Cloudflare, or just the provider blocking the ports for whatever reason. Was just interested to see if maybe there was a Cloudflare Tunnel explanation that we weren’t seeing.

Most likely not going to purse this further as it’s not actively impacting our infrastructure anymore, unless someone else responds with a similar issue. Thanks for your responses.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.