Argo Tunnel issues - error 1033

Hello,

I am trying to set up Argo Tunnel to secure access to internal services running on a Kubernetes cluster.

I have been following the Argo Tunnel Kubernetes guide, which puts cloudflared in front of an ingress controller (in my case, Istio), and it’s mostly working fine (after spending a few hours piecing together different documents).

I’m running into a few issues right now:

  1. When loading certain parts of websites I will get a 1033 error, the debug logs in the cloudflared pods still show traffic flowing, however. There are some 500 errors, but not enough information to debug with.

  2. The load balancer shows “tcp timeout” when being monitored via TCP/443, there is no obvious reason for this, as the cloudflared agents have 4 connections showing.

  3. Some sites absolutely refuse to load or lock up the browser for a significant amount of time. There is no clear reason for this. The page “loads” to a white screen and then just freezes.

  4. I’ve seen a single mention of enabling http2 via argo tunnels using the -p http2 flag, however when trying to run this with a named tunnel, I get an error that -p http2 is undefined, but the same command works fine on my local machine (with the exact same setup and same cloudflared version (2021.1.4)

Here is my current (somewhat working) config:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared01
  labels:
    app: cloudflared01
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cloudflared01
  template:
    metadata:
      labels:
        app: cloudflared01
    spec:
      containers:
      - name: cloudflared01
        image: docker.io/cloudflare/cloudflared:2021.1.4
        imagePullPolicy: Always
        command: ["cloudflared", "tunnel", "run"]
        args:
        # - --hostname=lb.XXXXXX.com
        # - --url=https://10.0.40.60
        # - --no-tls-verify
        # - --lb-pool=XXXXXX
        # - --origincert=/etc/cloudflared/cert.pem
        # - --config=/etc/cloudflared/config.yml
        # - --loglevel=debug
        # - --socks5
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        resources:
          limits:
            cpu: 100m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - mountPath: /etc/cloudflared/cert.pem
          name: tunnel-secret
          subPath: cert.pem
          readOnly: true
        - name: cloudflared-config
          mountPath: /etc/cloudflared/config.yml
          subPath: config.yaml
          readOnly: true
        - name: cloudflared-config
          mountPath: /etc/cloudflared/creds.json
          subPath: creds.json
          readOnly: true
        - name: tunnel-secret
          mountPath: /etc/cloudflared/cert.pem
          subPath: cert.pem
          readOnly: true
      terminationGracePeriodSeconds: 60
      dnsConfig:
        nameservers:
          - 10.0.40.20
          - 10.0.40.21
      volumes:
      - name: tunnel-secret
        secret:
          secretName: XXXXXX.com
      - name: cloudflared-config
        configMap:
          name: cloudflared01-config

and the config file:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared01-config
  namespace: default
data:
  config.yaml: |
    tunnel: cloudflared01
    credentials-file: /etc/cloudflared/creds.json
    origincert: /etc/cloudflared/cert.pem
    loglevel: debug
    prop-loglevel: debug
    ingress:
      - hostname: rancher.XXXXXX.com
        service: https://rancher.XXXXXX.XXXXXX.com
        originRequest:
          disableChunkedEncoding: true
          httpHostHeader: rancher.XXXXXX.XXXXXX.com
      - hostname: pve.XXXXXX.com
        service: https://pve.XXXXXX.XXXXXX.com:8006
        originRequest:
          disableChunkedEncoding: true
          httpHostHeader: rancher.XXXXXX.XXXXXX.com
      - service: https://10.0.40.60
        originRequest:
          noTLSVerify: true
  creds.json: |
  {"AccountTag":"XXXXXX","TunnelSecret":"XXXXXX","TunnelID":"XXXXXX","TunnelName":"cloudflared01"}
1 Like

Hi @justin3,

Thank you for using Argo Tunnel!

  1. The 1033 error means your tunnel wasn’t connected to our network at the time. You can run cloudflared tunnel list to see the list of connections from your tunnel.
  2. Are you using the cloudflare Load Balancer product?
  3. Do you know if the requests reached your tunnel, and was the tunnel able to connect with your service? Do you have network policies that can block connections from the tunnel pod to the service pod?
  4. In config.yaml, you can add protocol: http2 to enable the http2 transport.

I hope these will help!

2 Likes

Hello,

I have the following output:

cloudflared tunnel list
ID                NAME            CREATED                     CONNECTIONS
<removed> cloudflared01 2021-01-16T16:02:51Z 2xIAD, 2xORD

That tunnel has the following log:

6:18PM INF Starting tunnel tunnelID=<removed>
Sat, Feb 6 2021 1:18:04 pm	6:18PM INF Version 2021.1.4
Sat, Feb 6 2021 1:18:04 pm	6:18PM INF GOOS: linux, GOVersion: go1.15.6, GoArch: amd64
Sat, Feb 6 2021 1:18:04 pm	6:18PM INF Environment variables map[cred-file:/etc/cloudflared/creds.json credentials-file:/etc/cloudflared/creds.json p:http2 protocol:http2 proxy-dns-upstream:https://1.1.1.1/dns-query, https://1.0.0.1/dns-query]
Sat, Feb 6 2021 1:18:04 pm	6:18PM INF Autoupdate frequency is set autoupdateFreq=86400000
Sat, Feb 6 2021 1:18:04 pm	6:18PM INF Initial protocol http2
Sat, Feb 6 2021 1:18:04 pm	6:18PM INF Starting metrics server on 127.0.0.1:40417/metrics
Sat, Feb 6 2021 1:18:04 pm	6:18PM ERR update check failed: %s error="Get \"https://update.argotunnel.com?arch=amd64&os=linux\": dial tcp 104.18.7.49:443: connect: connection refused"
Sat, Feb 6 2021 1:18:06 pm	6:18PM DBG edgediscovery - GetAddr: Giving connection its new address address=198.41.200.13:7844 connIndex=0
Sat, Feb 6 2021 1:18:06 pm	6:18PM INF Retrying connection in 1s seconds error="DialContext error: dial tcp 198.41.200.13:7844: connect: connection refused" connIndex=0
Sat, Feb 6 2021 1:18:07 pm	6:18PM INF Retrying connection in 2s seconds error="DialContext error: dial tcp 198.41.200.13:7844: connect: connection refused" connIndex=0
Sat, Feb 6 2021 1:18:09 pm	6:18PM INF Retrying connection in 4s seconds error="DialContext error: dial tcp 198.41.200.13:7844: connect: connection refused" connIndex=0
Sat, Feb 6 2021 1:18:14 pm	6:18PM DBG Connecting via http2 connIndex=0
Sat, Feb 6 2021 1:18:14 pm	6:18PM INF Connection 49aae8c0-94ed-4f7c-8431-c1b52da65fd1 registered connIndex=0 location=ORD
Sat, Feb 6 2021 1:18:14 pm	6:18PM DBG edgediscovery - GetDifferentAddr: Giving connection its new address address=198.41.192.47:7844 connIndex=1
Sat, Feb 6 2021 1:18:14 pm	6:18PM DBG Connecting via http2 connIndex=1
Sat, Feb 6 2021 1:18:15 pm	6:18PM INF Connection d0eea3d5-6d51-4c81-9572-582cf5ca3d5b registered connIndex=1 location=IAD
Sat, Feb 6 2021 1:18:15 pm	6:18PM DBG edgediscovery - GetDifferentAddr: Giving connection its new address address=198.41.200.53:7844 connIndex=2
Sat, Feb 6 2021 1:18:15 pm	6:18PM DBG Connecting via http2 connIndex=2
Sat, Feb 6 2021 1:18:15 pm	6:18PM INF Connection 54fb35ae-3a1a-4752-a23f-9dbf0fadd7ce registered connIndex=2 location=ORD
Sat, Feb 6 2021 1:18:16 pm	6:18PM DBG edgediscovery - GetDifferentAddr: Giving connection its new address address=198.41.192.227:7844 connIndex=3
Sat, Feb 6 2021 1:18:16 pm	6:18PM DBG Connecting via http2 connIndex=3
Sat, Feb 6 2021 1:18:17 pm	6:18PM INF Connection b04e15b2-58da-413a-b089-9edb01b90d47 registered connIndex=3 location=IAD

I am using the Cloudflare Load Balancing product (I’m following this guide: https://developers.cloudflare.com/argo-tunnel/routing-to-tunnel/kubernetes#deploy-in-front-of-an-ingress-controller

When I specify the config via arguments, I can get the tunnels set up and working, however, when I set up the config via the config file and “cloudflared tunnel run”, it gives the above errors and I don’t think the tunnels are being created correctly. As far as I can tell, the container running cloudflared is able to reach the ingress gateway, though I cannot exec into the container to test.

I have added protocol: http2 to the config, which now looks like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared01-config
  namespace: default
data:
  config.yaml: |
    tunnel: cloudflared01
    credentials-file: /etc/cloudflared/creds.json
    loglevel: debug
    prop-loglevel: debug
    protocol: http2
    ingress:
      - hostname: rancher.justin-tech.com
        service: https://rancher.corp.justin-tech.com
        originRequest:
          disableChunkedEncoding: true
          httpHostHeader: rancher.corp.justin-tech.com
      - hostname: pve.justin-tech.com
        service: https://pve.corp.justin-tech.com:8006
        originRequest:
          disableChunkedEncoding: true
          httpHostHeader: pve.corp.justin-tech.com
      - service: https://10.0.40.60
        originRequest:
          noTLSVerify: true
  creds.json: |
    {"AccountTag":"","TunnelSecret":"","TunnelID":"","TunnelName":"cloudflared01"}

@justin3 Have you managed to solve this? I seem to have same issue. Works from command-line, but not from config.yml files…

Hello,

I vaguely recall making a change that forwarded all traffic to a reverse proxy and handled that part there.

Unfortunately, this environment no longer exists and I can’t find the config files that I used at the time.

Sorry I couldn’t be more help.