Argo Tunnel Kubernetes Sidecar Model

Zero Trust Cloudflare Tunnel
1

Attempting to follow https://developers.cloudflare.com/argo-tunnel/reference/sidecar/ documentation.

I’ve built the docker image that I’m using here: multi-arch-images/Dockerfile at main · raspbernetes/multi-arch-images · GitHub

I’m running it as a sidecar container in my deployment:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: “12”
helm.fluxcd.io/antecedent: kube-system:helmrelease/nginx-ingress
creationTimestamp: “2020-04-27T01:43:03Z”
generation: 13
labels:
app: nginx-ingress
app.kubernetes.io/component: controller
chart: nginx-ingress-1.36.3
heritage: Helm
release: nginx-ingress
name: nginx-ingress-controller
namespace: kube-system
resourceVersion: “3211025”
selfLink: /apis/apps/v1/namespaces/kube-system/deployments/nginx-ingress-controller
uid: b3042bb6-2846-4598-9ff9-ca699f43569c
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx-ingress
release: nginx-ingress
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx-ingress
app.kubernetes.io/component: controller
component: controller
release: nginx-ingress
spec:
containers:
- args:
- /nginx-ingress-controller
- --default-backend-service=kube-system/nginx-ingress-default-backend
- --election-id=ingress-controller-leader
- --ingress-class=nginx
- --configmap=kube-system/nginx-ingress-controller
- --default-ssl-certificate=kube-system/acme-crt-secret
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: Quay
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: nginx-ingress-controller
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
memory: 600Mi
requests:
cpu: 25m
memory: 500Mi
securityContext:
allowPrivilegeEscalation: true
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
runAsUser: 101
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
- args:
- --url=192.168.1.155
- --hostname=raspbernetes.com
- --origincert=/etc/cloudflared/cert.pem
- --no-autoupdate
- --proxy-connect-timeout=60s
- --proxy-tls-timeout=60s
- --no-tls-verify
- --loglevel=debug
command:
- cloudflared
- tunnel
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: raspbernetes/cloudflared:local
imagePullPolicy: IfNotPresent
name: cloudflared
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/cloudflared
name: tunnel-secret
readOnly: true
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: nginx-ingress
serviceAccountName: nginx-ingress
terminationGracePeriodSeconds: 60
volumes:
- name: tunnel-secret
secret:
defaultMode: 420
secretName: raspbernetes.com-cloudflared-cert
status:
availableReplicas: 2
conditions:

  • lastTransitionTime: “2020-04-27T01:43:03Z”
    lastUpdateTime: “2020-04-27T13:50:36Z”
    message: ReplicaSet “nginx-ingress-controller-775f57c6f5” has successfully progressed.
    reason: NewReplicaSetAvailable
    status: “True”
    type: Progressing
  • lastTransitionTime: “2020-04-27T13:53:46Z”
    lastUpdateTime: “2020-04-27T13:53:46Z”
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: “True”
    type: Available
    observedGeneration: 13
    readyReplicas: 2
    replicas: 2
    updatedReplicas: 2

I am successfully able to run my main container without the sidecar cloudflared container. I have also ran the cloudflared daemon on my node outside my kubernetes cluster and configured it to work with my static IP and that worked.

However, when running it within the container inside my cluster it would not work.

Errors look along these lines:

logs

time=“2020-04-27T13:50:38Z” level=warning msg=“Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.Cloudflare-warp ~/Cloudflare-warp /usr/local/etc/cloudflared /etc/cloudflared]”
time=“2020-04-27T13:50:38Z” level=warning msg=“At debug level, request URL, method, protocol, content legnth and header will be logged. Response status, content length and header will also be logged in debug level.”
time=“2020-04-27T13:50:38Z” level=info msg=“Version 2020.4.0”
time=“2020-04-27T13:50:38Z” level=info msg=“GOOS: linux, GOVersion: go1.14.2, GoArch: arm64”
time=“2020-04-27T13:50:38Z” level=info msg=Flags hostname=raspbernetes.com loglevel=debug no-autoupdate=true no-tls-verify=true origincert=/etc/cloudflared/cert.pem proxy-connect-timeout=1m0s proxy-dns-upstream=“https://1.1.1.1/dns-query, https://1.0.0.1/dns-query” proxy-tls-timeout=1m0s url=192.168.1.155
time=“2020-04-27T13:50:39Z” level=info msg=“Starting metrics server” addr=“127.0.0.1:39611/metrics”
time=“2020-04-27T13:51:36Z” level=info msg=“Proxying tunnel requests to http://192.168.1.155”
time=“2020-04-27T13:51:38Z” level=debug msg=“Giving connection its new address” connID=0 function=GetAddr subsystem=edgediscovery
time=“2020-04-27T13:51:53Z” level=error msg=“Tunnel creation failure” connectionID=0 error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”
time=“2020-04-27T13:51:53Z” level=error msg=“Quitting due to error” error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”
time=“2020-04-27T13:51:53Z” level=info msg=“Metrics server stopped”

If anyone has seen success in running Cloudflare Tunnel with the Kubernetes Sidecar approach I’d like to hear about how it went, and potentially see what configuration was required to get it to work.

2

Hi Michael,

In this case - can you try changing your hostname parameter to a subdomain? The error message you get is quite vague, but Argo Tunnels only operates on subdomains:

I’m curious to see if it behaves differently when change the hostname to a subdomain.

3

Attempted running on a subdomain and no luck.

Logs are below:

logs

❯ k logs nginx-ingress-controller-7f4696c794-n6mkx cloudflared
time=“2020-04-30T10:01:51Z” level=warning msg=“Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.Cloudflare-warp ~/Cloudflare-warp /usr/local/etc/cloudflared /etc/cloudflared]”
time=“2020-04-30T10:01:51Z” level=warning msg=“At debug level, request URL, method, protocol, content legnth and header will be logged. Response status, content length and header will also be logged in debug level.”
time=“2020-04-30T10:01:51Z” level=info msg=“Version 2020.4.0”
time=“2020-04-30T10:01:51Z” level=info msg=“GOOS: linux, GOVersion: go1.14.2, GoArch: arm64”
time=“2020-04-30T10:01:52Z” level=info msg=Flags hostname=ingress.raspbernetes.com loglevel=debug no-autoupdate=true no-tls-verify=true origincert=/etc/cloudflared/cert.pem proxy-connect-timeout=1m0s proxy-dns-upstream=“https://1.1.1.1/dns-query, https://1.0.0.1/dns-query” proxy-tls-timeout=1m0s url=“127.0.0.1:80”
time=“2020-04-30T10:01:53Z” level=info msg=“Starting metrics server” addr=“127.0.0.1:46841/metrics”
❯ k logs nginx-ingress-controller-7f4696c794-n6mkx cloudflared -f
time=“2020-04-30T10:01:51Z” level=warning msg=“Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.Cloudflare-warp ~/Cloudflare-warp /usr/local/etc/cloudflared /etc/cloudflared]”
time=“2020-04-30T10:01:51Z” level=warning msg=“At debug level, request URL, method, protocol, content legnth and header will be logged. Response status, content length and header will also be logged in debug level.”
time=“2020-04-30T10:01:51Z” level=info msg=“Version 2020.4.0”
time=“2020-04-30T10:01:51Z” level=info msg=“GOOS: linux, GOVersion: go1.14.2, GoArch: arm64”
time=“2020-04-30T10:01:52Z” level=info msg=Flags hostname=ingress.raspbernetes.com loglevel=debug no-autoupdate=true no-tls-verify=true origincert=/etc/cloudflared/cert.pem proxy-connect-timeout=1m0s proxy-dns-upstream=“https://1.1.1.1/dns-query, https://1.0.0.1/dns-query” proxy-tls-timeout=1m0s url=“127.0.0.1:80”
time=“2020-04-30T10:01:53Z” level=info msg=“Starting metrics server” addr=“127.0.0.1:46841/metrics”
time=“2020-04-30T10:03:01Z” level=info msg=“Proxying tunnel requests to http://127.0.0.1:80”
time=“2020-04-30T10:03:03Z” level=debug msg=“Giving connection its new address” connID=0 function=GetAddr subsystem=edgediscovery
time=“2020-04-30T10:03:17Z” level=error msg=“Tunnel creation failure” connectionID=0 error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”
time=“2020-04-30T10:03:17Z” level=info msg=“Metrics server stopped”
time=“2020-04-30T10:03:18Z” level=error msg=“Quitting due to error” error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”

4

OK - couple of diagnostic steps to run from inside your container where the daemon is running:

dig region1.argotunnel.com
dig region2.argotunnel.com

And then:

openssl s_client -connect region1.argotunnel.com:7844
openssl s_client -connect region2.argotunnel.com:7844

Or:

curl -k https://region1.argotunnel.com:7844 -sv
curl -k https://region2.argotunnel.com:7844 -sv
5

Just updated my image with those dev tools and here is the output, hopefully we can figure out this issue.

$ dig region1.argotunnel.com

logs

; <<>> DiG 9.14.8 <<>> region1.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54248
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;region1.argotunnel.com. IN A

;; ANSWER SECTION:
region1.argotunnel.com. 30 IN A 198.41.192.227
region1.argotunnel.com. 30 IN A 198.41.192.7
region1.argotunnel.com. 30 IN A 198.41.192.47
region1.argotunnel.com. 30 IN A 198.41.192.167
region1.argotunnel.com. 30 IN A 198.41.192.107

;; Query time: 99 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Thu Apr 30 10:48:55 UTC 2020
;; MSG SIZE rcvd: 241

$ dig region2.argotunnel.com

logs

; <<>> DiG 9.14.8 <<>> region2.argotunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 45851
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;region2.argotunnel.com. IN A

;; ANSWER SECTION:
region2.argotunnel.com. 30 IN A 198.41.200.13
region2.argotunnel.com. 30 IN A 198.41.200.113
region2.argotunnel.com. 30 IN A 198.41.200.233
region2.argotunnel.com. 30 IN A 198.41.200.53
region2.argotunnel.com. 30 IN A 198.41.200.193

;; Query time: 298 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Thu Apr 30 10:49:32 UTC 2020
;; MSG SIZE rcvd: 24

$ curl -k https://region1.argotunnel.com:7844 -sv

logs
  • Trying 198.41.192.47:7844…
  • TCP_NODELAY set
  • Connected to region1.argotunnel.com (198.41.192.47) port 7844 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server did not agree to a protocol
  • Server certificate:
  • subject: O=Cloudflare, Inc.; OU=Cloudflare Origin CA; CN=Cloudflare Origin Certificate
  • start date: Jul 30 19:51:00 2018 GMT
  • expire date: Jul 26 19:51:00 2033 GMT
  • issuer: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; OU=Cloudflare Origin SSL ECC Certificate Authority
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

GET / HTTP/1.1
Host: region1.argotunnel.com:7844
User-Agent: curl/7.67.0
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

  • Received HTTP/0.9 when not allowed

  • Closing connection 0

  • TLSv1.3 (OUT), TLS alert, close notify (256):

$ curl -k https://region2.argotunnel.com:7844 -sv

logs
  • Trying 198.41.200.13:7844…
  • TCP_NODELAY set
  • Connected to region2.argotunnel.com (198.41.200.13) port 7844 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: none
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server did not agree to a protocol
  • Server certificate:
  • subject: O=Cloudflare, Inc.; OU=Cloudflare Origin CA; CN=Cloudflare Origin Certificate
  • start date: Jul 30 19:51:00 2018 GMT
  • expire date: Jul 26 19:51:00 2033 GMT
  • issuer: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; OU=Cloudflare Origin SSL ECC Certificate Authority
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

GET / HTTP/1.1
Host: region2.argotunnel.com:7844
User-Agent: curl/7.67.0
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

  • Received HTTP/0.9 when not allowed

  • Closing connection 0

  • TLSv1.3 (OUT), TLS alert, close notify (256):

6

@simon any luck in reviewing my output from the commands you asked me to perform?

7

@michaelfornaro your output there shows that you can connect, so there’s something unusual happening here. In the absence of anything else, I’d suggest a PCAP while attempting to start the tunnel to understand exactly what is happening - that will at least show us what the client is doing to see if we can unpick it.

8

Is it possible that there could be some type of exotic race condition where my container with the service hasn’t started and the tunnel is trying to establish before it is available?

9

Very difficult to say at this point - I’d recommend getting a PCAP so we can understand what’s happening on the wire.

10

@simon I feel a little out of my expertise with this specific issue. I attempted updating the cloudflared client to 2020.5.1 with no such luck.

Logs

❯ k logs nginx-ingress-controller-79c58f599f-r4nfq -c cloudflared -f
time=“2020-05-27T06:49:28Z” level=warning msg=“Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.Cloudflare-warp ~/Cloudflare-warp /usr/local/etc/cloudflared /etc/cloudflared]”
time=“2020-05-27T06:49:28Z” level=warning msg=“At debug level, request URL, method, protocol, content legnth and header will be logged. Response status, content length and header will also be logged in debug level.”
time=“2020-05-27T06:49:29Z” level=info msg=“Version 2020.5.1-6-ga908453”
time=“2020-05-27T06:49:29Z” level=info msg=“GOOS: linux, GOVersion: go1.14.3, GoArch: arm64”
time=“2020-05-27T06:49:29Z” level=info msg=Flags hostname=test.raspbernetes.com loglevel=debug no-autoupdate=true origincert=/etc/cloudflared/cert.pem proxy-dns-upstream=“https://1.1.1.1/dns-query, https://1.0.0.1/dns-query” url=“127.0.0.1:80”
time=“2020-05-27T06:49:32Z” level=debug msg=“Reading origin cert from /etc/cloudflared/cert.pem”
time=“2020-05-27T06:49:32Z” level=info msg=“Starting metrics server” addr=“127.0.0.1:38155/metrics”
time=“2020-05-27T06:49:54Z” level=info msg=“Proxying tunnel requests to http://127.0.0.1:80”
time=“2020-05-27T06:50:04Z” level=debug msg=“Giving connection its new address 198.41.200.113:7844” connID=0 function=GetAddr subsystem=edgediscovery
time=“2020-05-27T06:50:23Z” level=error msg=“Tunnel creation failure” connectionID=0 error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”
time=“2020-05-27T06:50:23Z” level=info msg=“Initiating graceful shutdown due to h2mux handshake with edge error: Handshake error: 1000 handshake timeout …”
time=“2020-05-27T06:50:23Z” level=error msg=“Quitting due to error” error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”
time=“2020-05-27T06:50:24Z” level=info msg=“Metrics server stopped”
h2mux handshake with edge error: Handshake error: 1000 handshake timeout