I am having some trouble using a cloudflared tunnel to connect to my kubernetes clusters. I have multiple existing k8s clusters hosted in AWS EKS with cloudflared running, and tunnels in each cluster that are currently routing to various http services, all of which are working as expected. However, when following the instructions in this article provided by cloudflare, I am seeing the error:
ERR failed to connect to origin error='websocket: bad handshake' originURL=https://redacted.ai when attempting to connect via a client machine.
Steps to reproduce the behavior:
I have followed all of the steps in the document provided above. From the start, I have:
- Created a zero trust policy for my application
- Installed the cloudflared deployment on my cluster
- Created and configured a tunnel using ingress rules
- Created a DNS record to route traffic to the tunnel
- Run the tunnel
The last step, attempting to connect via
cloudflared access tcp , brings me to the error mentioned above. My ingress looks like this:
tunnel: redacted credentials-file: /etc/cloudflared/creds/cloudflared metrics: 0.0.0.0:2000 no-autoupdate: true ingress: - hostname: redacted service: redacted (ROUTES TO HTTP SERVICE, WORKING CORRECTLY) - hostname: redacted service: redacted (ROUTES TO HTTP SERVICE, WORKING CORRECTLY) - hostname: test-url.redacted.ai service: tcp://kubernetes.docker.internal:6443 originRequest: noTLSverify: true proxyType: socks # any traffic which didn't match a previous rule, and responds with HTTP 404. - service: http_status:404
Attempting to connect via another machine looks like this:
create connection to Cloudflare:
cloudflared access tcp --hostname test-url.redacted.ai --url 127.0.0.1:8080
Then in another terminal window:
env HTTPS_PROXY=socks5://127.0.0.1:8080 kubectl get po
Expected behavior is to have the ability to run kubectl commands via this configuration.
Environment and versions
Local machine attempting to connect is MacOS Montery v12.1. Kubernetes clusters are hosted in AWS using EKS managed node groups.
Logs and errors
Aside from the errors mentioned above, I can see these logs from the cloudflared pod:
ERR error="dial tcp: lookup kubernetes.docker.internal on 172.20.0.10:53: no such host" cfRay=7149a9a43f5d7dd2-LAX ingressRule=2 originService=tcp://kubernetes.docker.internal:6443 ERR Failed to handle QUIC stream error="dial tcp: lookup kubernetes.docker.internal on 172.20.0.10:53: no such host" connIndex=2
172.20.0.10:53 I believe corresponds to the default service kubernetes.default.svc.cluster.local.
Any help with this is greatly appreciated, as the Cloudflare docs are limited and there doesn’t seem to be much information about this particular issue online.