Feedback for Cloudflare Access' support for kubectl

The Cloudflare team is exicted to announce support for kubectl in Cloudflare Access.

Starting today, you can use Cloudflare Access and Argo Tunnel to securely manage your Kubernetes cluster with the kubectl command-line tool.

We built this to address one of the edge cases that stopped all of Cloudflare, as well as some of our customers, from disabling the VPN. With this workflow, you can add SSO requirements and a zero-trust model to your Kubernetes management in under 30 minutes.

We’d like your feedback! Documentation to get started is available below. Please feel free to use this thread for any feedback (good, bad, or otherwise) to help us make the feature even better.

1 Like

Feedback: Has been a horrible experience, still haven’t successfully had this working - I’ve been facing multiple issues and have still got an open support ticket almost 18 days later. https://support.cloudflare.com/hc/en-us/requests/1865448

Hey Michael, sorry to hear about that. I’ll ask the support team to take a look at that ticket. Anything you can share here that we could help troubleshoot?

I’ve configured the cloudflared on the node that has access to the ILB of the kube-apiserver (this is the server address also used in my kubeconfig which works whilst on the network). I can see it create the AAAA dns records in cloudflare and that all seems fine (or so i think).

I then run the following commands with the resulting error:

❯ cloudflared access tcp --hostname api-server.raspbernetes.com --url 127.0.0.3:1234
INFO[0000] Start Websocket listener on: 127.0.0.3:1234
❯ env HTTPS_PROXY=socks5://127.0.0.3:1234 kubectl get pods
Unable to connect to the server: proxyconnect tcp: dial tcp 127.0.0.3![1234|20pxx20px](https://static.zdassets.com/classic/images/emojis/1234.png "1234") i/o timeout

I can also see no traffic reaching the daemon running on the node from a cloudflare or kube-apiserver perspective - there also doesn’t appear to be much more logging I can put onto the cloudflared command.

I’ve set the kubectl -v=8 and it all looks fairly the same with these reoccurring timeouts. There were also issues that have been resolved regarding the CLI not having the required flags or commands which has since been released on the latest version.

Any insights would be awesome!!

Hey Michael, definitely odd. It looks like your system is treating the port number (:1234) as an emoji? Could you try with a different port number that doesn’t map to an emoji just to rule that out?

The emoji was just a copy and paste and that was how it rendered here in this message. Either way I tried another port and can confirm it is still the same error unfortunately.

Thanks for the clarification.
Do you have a host file for 127.0.0.3? That is used as an example in the docs, but unless you have a dedicated file for that, could you try using 127.0.0.1 or localhost? That needs to be a loopback address.

Brilliant, that works! Thanks for your help, I’m curious why the docs use the 127.0.0.3 in the example, it’s not a default /etc/host entry. It does make a lot of sense that of-course that would not work.

I’ve finally got my last outstanding issue here Argo Tunnel Kubernetes Sidecar Model

Would love your feedback on that thread, that should conclude all my Kubernetes integration.

Hi there,
I am going to test Cloudflare access with kubectl but a question raise in my mind.

How can the Cloudflare Access workflow be applied to a pipeline run, like CircleCI SaaS service.
There I need a way to pragmatically pass the credential without any human interaction.

Any idea on how to achieve it?
Thanks & Best Regards
Alessandro

@SamRhea I’m also curious because I want to run my service in an automated fashion, I don’t see anywhere in the configuration here https://developers.cloudflare.com/argo-tunnel/reference/arguments/ where I can set the --socks5=true argument. Is this something that can be added? How would you recommend starting the service in the meanwhile without this being an option?

Couple topics here.

1) Automated systems
@alessandro.surace Service tokens can be used here, just issue those to the calling service

2) Running kubectl as a service
You can specify that in the cloudflared config file, it’s just not listed in the arguments docs yet.

1 Like

@SamRhea which option should be added?

Thanks, good to know it’s supported in the config file.

I’m running a tunnel to my cluster ingress and one to my kube-apiserver

I’m getting this error on the 2nd tunnel that starts

{"error":"already connected to this server","level":"warning","msg":"Tunnel disconnected due to error","time":"2020-05-11T18:44:51+10:00"}

My ingress tunnel is setup to my URL example.com whereas my kube-apiserver is setup to api-server.example.com and I have a CNAME that is FROM * TO raspbernetes.com.

That error is expected. You cannot run multiple tunnels in the same instance of cloudflared when using the config file to start it as a service. You’ll need to run a second instance and ensure it is pointing to a different port.

I was running cloudflared on two seperate machines for seperate hosts, not sure how that seemed to have collided but I’ll attempt again tomorrow.

I’m also having issues with the following:

Setting up my CNAME to route FROM “*” TO “@”.

EG. I have my ingress controller running on “example.com” and I want to have a CNAME with “.example.com" to redirect to “example.com” this use to be achievable - I would need to enable the CNAME to be proxied to avoid leaking the resolvable host IP but now when I enter the "” wildcard into the CNAME it says “DNS only” and won’t allow me to proxy the CNAME?

Secondly, I’ve Attempting to follow https://developers.cloudflare.com/argo-tunnel/reference/sidecar/ documentation.

I’ve built the docker image that I’m using here: https://github.com/raspbernetes/multi-arch-images/blob/master/build/cloudflared/Dockerfile

I’m running it as a sidecar container in my deployment:

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: “12”
helm.fluxcd.io/antecedent: kube-system:helmrelease/nginx-ingress
creationTimestamp: “2020-04-27T01:43:03Z”
generation: 13
labels:
app: nginx-ingress
app.kubernetes.io/component: controller
chart: nginx-ingress-1.36.3
heritage: Helm
release: nginx-ingress
name: nginx-ingress-controller
namespace: kube-system
resourceVersion: “3211025”
selfLink: /apis/apps/v1/namespaces/kube-system/deployments/nginx-ingress-controller
uid: b3042bb6-2846-4598-9ff9-ca699f43569c
spec:
progressDeadlineSeconds: 600
replicas: 2
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx-ingress
release: nginx-ingress
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx-ingress
app.kubernetes.io/component: controller
component: controller
release: nginx-ingress
spec:
containers:

  • args:
  • /nginx-ingress-controller
  • –default-backend-service=kube-system/nginx-ingress-default-backend
  • –election-id=ingress-controller-leader
  • –ingress-class=nginx
  • –configmap=kube-system/nginx-ingress-controller
  • –default-ssl-certificate=kube-system/acme-crt-secret
    env:
  • name: POD_NAME
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: metadata.name
  • name: POD_NAMESPACE
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: metadata.namespace
    image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller-arm:0.30.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
    failureThreshold: 3
    httpGet:
    path: /healthz
    port: 10254
    scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
    name: nginx-ingress-controller
    ports:
  • containerPort: 80
    name: http
    protocol: TCP
  • containerPort: 443
    name: https
    protocol: TCP
    readinessProbe:
    failureThreshold: 3
    httpGet:
    path: /healthz
    port: 10254
    scheme: HTTP
    initialDelaySeconds: 10
    periodSeconds: 10
    successThreshold: 1
    timeoutSeconds: 1
    resources:
    limits:
    memory: 600Mi
    requests:
    cpu: 25m
    memory: 500Mi
    securityContext:
    allowPrivilegeEscalation: true
    capabilities:
    add:
  • NET_BIND_SERVICE
    drop:
  • ALL
    runAsUser: 101
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  • args:
  • –url=192.168.1.155
  • –hostname=raspbernetes.com
  • –origincert=/etc/cloudflared/cert.pem
  • –no-autoupdate
  • –proxy-connect-timeout=60s
  • –proxy-tls-timeout=60s
  • –no-tls-verify
  • –loglevel=debug
    command:
  • cloudflared
  • tunnel
    env:
  • name: POD_NAME
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: metadata.name
  • name: POD_NAMESPACE
    valueFrom:
    fieldRef:
    apiVersion: v1
    fieldPath: metadata.namespace
    image: raspbernetes/cloudflared:local
    imagePullPolicy: IfNotPresent
    name: cloudflared
    resources:
    limits:
    cpu: 10m
    memory: 20Mi
    requests:
    cpu: 10m
    memory: 20Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
  • mountPath: /etc/cloudflared
    name: tunnel-secret
    readOnly: true
    dnsPolicy: ClusterFirst
    restartPolicy: Always
    schedulerName: default-scheduler
    securityContext: {}
    serviceAccount: nginx-ingress
    serviceAccountName: nginx-ingress
    terminationGracePeriodSeconds: 60
    volumes:
  • name: tunnel-secret
    secret:
    defaultMode: 420
    secretName: raspbernetes.com-cloudflared-cert

I am successfully able to run my main container without the sidecar cloudflared container. I have also ran the cloudflared daemon on my node outside my kubernetes cluster and configured it to work with my static IP and that worked.

However, when running it within the container inside my cluster it would not work.

Errors look along these lines:

logs

time=“2020-04-27T13:50:38Z” level=warning msg=“Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /usr/local/etc/cloudflared /etc/cloudflared]”
time=“2020-04-27T13:50:38Z” level=warning msg=“At debug level, request URL, method, protocol, content legnth and header will be logged. Response status, content length and header will also be logged in debug level.”
time=“2020-04-27T13:50:38Z” level=info msg=“Version 2020.4.0”
time=“2020-04-27T13:50:38Z” level=info msg=“GOOS: linux, GOVersion: go1.14.2, GoArch: arm64”
time=“2020-04-27T13:50:38Z” level=info msg=Flags hostname=raspbernetes.com loglevel=debug no-autoupdate=true no-tls-verify=true origincert=/etc/cloudflared/cert.pem proxy-connect-timeout=1m0s proxy-dns-upstream=“https://1.1.1.1/dns-query, https://1.0.0.1/dns-query” proxy-tls-timeout=1m0s url=192.168.1.155
time=“2020-04-27T13:50:39Z” level=info msg=“Starting metrics server” addr=“127.0.0.1:39611/metrics”
time=“2020-04-27T13:51:36Z” level=info msg=“Proxying tunnel requests to http://192.168.1.155
time=“2020-04-27T13:51:38Z” level=debug msg=“Giving connection its new address” connID=0 function=GetAddr subsystem=edgediscovery
time=“2020-04-27T13:51:53Z” level=error msg=“Tunnel creation failure” connectionID=0 error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”
time=“2020-04-27T13:51:53Z” level=error msg=“Quitting due to error” error=“h2mux handshake with edge error: Handshake error: 1000 handshake timeout”
time=“2020-04-27T13:51:53Z” level=info msg=“Metrics server stopped”

I have my suspicions this may be some type of exotic race condition between the sidecar and the service starting and exposing the service. I’m not sure if cloudflared in the sidecar approach is attempting to bind to the service before it has become available?

Any insights with these would be welcomed as I’m sure others will have similar issues going forward.

@SamRhea any thoughts on my previous comment?

Hi Michael,

Tagging in the DNS team on the CNAME question; that should be functional but I’ll let @dina address.

On the sidecar, glad it’s working on the main container. I think you’re right that there is a liveness check issue with the sidecar coming online first. Can you continue to run it alongside the cluster?

HI @michaelfornaro,
Did you solve your problem?
I see your error and it seem related to some Cloudflare Edge problem.
In the example below I did a test with a specific cloudflare daemon version: 2020.4.0 . May be it makes a difference.

I’d been successful in creating a tunnel with the sidecar model with the hello_world example.
Let me share my configuration that may be can help you.

apiVersion: v1
kind: Service
metadata:
labels:
app: hello
name: hello
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort:
# targetPort defaults to the port configured above if left blank.
selector:
app: hello

apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: hello
name: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
template:
metadata:
labels:
app: hello
spec:
containers:
- name: hello
image: k8s.gcr.io/echoserver:1.4
#image: gcr.io/google-samples/gb-frontend:v4
imagePullPolicy: Always
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
ports:
- containerPort: 80
resources:
limits:
cpu: 100m
memory: 40Mi
requests:
cpu: 20m
memory: 40Mi
- name: tunnel
image: zioalex/cloudflared:2020.4.0
imagePullPolicy: IfNotPresent
command: [“cloudflared”, “tunnel”]
args:
- --url=http://127.0.0.1:8080
#- --url=http://172.20.152.63:8080
- --hostname=MY_HOSTNAME
- --origincert=/etc/cloudflared/cert.pem
- --loglevel=debug
- --no-autoupdate
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
volumeMounts:
- mountPath: /etc/cloudflared
name: tunnel-secret
readOnly: true
terminationGracePeriodSeconds: 60
volumes:
- name: tunnel-secret
secret:
secretName: MY_SECRET

Cheers

Interestingly enough that worked when i used an arm64 based echo server.

The issue I have is I’m attempting to use cloudflared to sidecar my nginx-ingress-controller to front all traffic, and that is what is giving the errors.

This is currently how I’m using it as I’m using the helm chart.

controller:
  extraContainers:
    - name: cloudflared
      image: raspbernetes/cloudflared:48b097a763db0cb8e8222057a4e6fe95775376ac
      imagePullPolicy: Always
      command: ['cloudflared', 'tunnel']
      args:
        - --url=http://127.0.0.1:30243
        - --hostname=test.raspbernetes.com
        - --origincert=/etc/cloudflared/cert.pem
        - --no-autoupdate
        - --loglevel=debug
      env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      resources:
        limits:
          cpu: 10m
          memory: 20Mi
        requests:
          cpu: 10m
          memory: 20Mi
      volumeMounts:
        - mountPath: /etc/cloudflared
          name: tunnel-secret
          readOnly: true
  extraVolumes:
    - name: tunnel-secret
      secret:
        secretName: raspbernetes.com-cloudflared-cert