Hello!
We’re excited to get your thoughts and ideas about Cloudflare Ingress Controller beta. Let us know what you think - both good and bad, and how you end up using the product. Looking forward to hearing from you.
To get started with The Cloudflare Ingress Controller, check out the docs at https://warp.Cloudflare.com/docs/kubernetes/.
The Cloudflare Ingress Controller is open source. You can check out the source at https://github.com/Cloudflare/Cloudflare-warp-ingress
-Dani
Hi Dani,
I’m getting the following error in my logs:
E1204 10:27:09.552944 1 reflector.go:201] github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:298: Failed to list *v1beta1.Ingress: User "system:serviceaccount:default:default" cannot list ingresses.extensions in the namespace "default". (get ingresses.extensions)
I1204 10:27:10.437885 1 reflector.go:236] Listing and watching *v1.Service from github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:296
I1204 10:27:10.441416 1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/default/services?resourceVersion=0 403 Forbidden in 3 milliseconds
Edit: Ok, seems the default service account does not have permissions. I fixed it with a temporary fix.
Now, I am still hitting some wall
I1204 14:12:50.391457 1 reflector.go:236] Listing and watching *v1beta1.Ingress from github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:298
I1204 14:12:50.392424 1 reflector.go:236] Listing and watching *v1.Endpoints from github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:297
I1204 14:12:50.395058 1 round_trippers.go:405] GET https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses?resourceVersion=0 200 OK in 3 milliseconds
I1204 14:12:50.405573 1 controller.go:149] Annotation kubernetes.io/ingress.class=Cloudflare-warp
I1204 14:12:50.405931 1 shared_informer.go:116] caches populated
I1204 14:12:50.414974 1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/default/endpoints?resourceVersion=0 200 OK in 22 milliseconds
I1204 14:12:50.419445 1 round_trippers.go:405] GET https://10.96.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses?resourceVersion=11968039&timeoutSeconds=481&watch=true 200 OK in 14 milliseconds
I1204 14:12:50.423745 1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/default/endpoints?resourceVersion=11978521&timeoutSeconds=582&watch=true 200 OK in 4 milliseconds
I1204 14:12:50.580593 1 reflector.go:236] Listing and watching *v1.Service from github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:296
I1204 14:12:50.586572 1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/default/services?resourceVersion=0 200 OK in 5 milliseconds
I1204 14:12:50.592026 1 round_trippers.go:405] GET https://10.96.0.1:443/api/v1/namespaces/default/services?resourceVersion=11968030&timeoutSeconds=499&watch=true 200 OK in 3 milliseconds
I1204 14:12:50.606175 1 shared_informer.go:116] caches populated
I1204 14:12:50.606978 1 controller.go:505] creating tunnel for ingress warp-ingress, key default/warp-ingress
I1204 14:12:50.608550 1 controller.go:541] Start or Stop default/warp-ingress
I1204 14:12:50.608631 1 controller.go:567] Validation ok for starting warp-service/warp-service/1
I1204 14:12:50.608699 1 controller.go:491] Error processing add:default/warp-ingress: Not implemented
I1204 14:12:50.613817 1 controller.go:362] Tunnel "default/warp-ingress" (warp-on-k8s.example.com) already exists
time="2017-12-04T14:12:52Z" level=info msg="Connected to AMS"
time="2017-12-04T14:12:52Z" level=info msg="There are currently 0 active tunnels for this zone. You are allowed to have 10" subsystem=rpc
time="2017-12-04T14:12:52Z" level=info msg="Registered at https://warp-on-k8s.example.com"
time="2017-12-04T14:12:52Z" level=info msg="There are currently 0 active tunnels for this zone. You are allowed to have 10"
I1204 14:13:05.392118 1 reflector.go:276] github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:298: forcing resync
I1204 14:13:05.392218 1 controller.go:149] Annotation kubernetes.io/ingress.class=Cloudflare-warp
I1204 14:13:05.392766 1 reflector.go:276] github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:297: forcing resync
I1204 14:13:05.392829 1 controller.go:283] Watching endpoint default/warp-service
I1204 14:13:05.392860 1 controller.go:541] Start or Stop default/warp-service
I1204 14:13:05.392870 1 controller.go:491] Error processing update:default/warp-service: Tunnel not found for key default/warp-service
I1204 14:13:05.398076 1 controller.go:541] Start or Stop default/warp-service
I1204 14:13:05.398119 1 controller.go:491] Error processing update:default/warp-service: Tunnel not found for key default/warp-service
I1204 14:13:05.408278 1 controller.go:541] Start or Stop default/warp-service
I1204 14:13:05.408303 1 controller.go:491] Error processing update:default/warp-service: Tunnel not found for key default/warp-service
I1204 14:13:05.428567 1 controller.go:541] Start or Stop default/warp-service
I1204 14:13:05.428607 1 controller.go:491] Error processing update:default/warp-service: Tunnel not found for key default/warp-service
I1204 14:13:05.468899 1 controller.go:541] Start or Stop default/warp-service
I1204 14:13:05.468936 1 controller.go:491] Error processing update:default/warp-service: Tunnel not found for key default/warp-service
I1204 14:13:05.549229 1 controller.go:541] Start or Stop default/warp-service
E1204 14:13:05.549269 1 controller.go:500] Dropping object "update:default/warp-service" out of the queue: Tunnel not found for key default/warp-service
I1204 14:13:05.581091 1 reflector.go:276] github.com/Cloudflare/Cloudflare-warp-ingress/pkg/controller/controller.go:296: forcing resync
I1204 14:13:05.581203 1 controller.go:216] Watching service default/warp-service
I1204 14:13:05.581235 1 controller.go:541] Start or Stop default/warp-service
I1204 14:13:05.581249 1 controller.go:491] Error processing update:default/warp-service: Tunnel not found for key default/warp-service
@entryninja The log level is set quite high, so there’s a lot of messages there which are probably not actually errors. We need to clean those up a little. With respect to RBAC configuration, the role currently requires these rules,
rules:
- apiGroups:
- ""
resources:
- secrets
verbs:
- get
- apiGroups:
- ""
- "extensions"
resources:
- ingresses
- services
- endpoints
verbs:
- list
- get
- watch
so, services and endpoints as well as ingresses. I think you have that configured correctly. Do you have pods and endpoints behind your service? If so, the log should later say something about there being 1 active tunnel et cetera.
Thanks for the reply!
I have followed the exact guide as specified in the opening post. So I have the httpbin deployment, the service, the ingress and the warp deployment running the actual warp.
That 1 active is never showing up. Not sure why. It keeps looping on the Tunnel not found for key default/warp-service.
How would you like me to set the output log? I’m running a clean install of K8s 1.7.3
I’ve made a small change that may help your setup. I need to do a little bit of testing with it right now. If you’d like to try it too 
you can change the image tag in your deployment manifest from quay.io/stackpoint/warp-controller:0.1.1 to quay.io/stackpoint/warp-controller:beta
update/edit: not quite there yet
So, a couple of things. I’ve updated parts of the code to improve error handling and logging, and to get that, you’ll want to change the deployment spec to read
image: quay.io/stackpoint/warp-controller:beta
imagePullPolicy: Always
More important !!, there’s an undocumented assumption that the name of the ingress and the service must be the same. We need to update the examples in the documentation to match this. There are three places in the yaml that should be changed. For now, set the name of the service,
- kind: Service
apiVersion: v1
metadata:
name: httpbin
To match the name of the ingress
- apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: httpbin
and the service the ingress refers to
backend:
serviceName: httpbin
To be frank, this is a bug, and will be fixed. The controller should not require the names to match.
Updated using your code and it works like a charm
Edit: only for so long 
After running for about 6 hours, I am getting a Cloudflare HTTP 526 error page (Invalid SSL certificate).
Is this related to the ingress or a bug in Warp itself?
The instructions on https://warp.Cloudflare.com/docs/kubernetes/
are slightly different than the objects in the github repo under deploy/
Which ones are correct?
I am finding the warp-controller pretty much kills my minikube VM. Bug, or am I doing something wrong?
I’ve tested on minikube without seeing excessive cpu or memory usage – in theory the resource limits on the pod should constrain it pretty well, and in practice, looking at heapster statistics, it’s been ok for me. But I’m happy to look into it more if you are seeing problems.
Just wanted to leave some positive feedback… The ingress controller appears to work great in our test environment on an Azure-managed K8s cluster! We are currently exposing three web services on different subdomains from our cluster and have successfully removed all public IP addresses. The transition from our own Docker image and multiple containers running the Warp client to the K8s Ingress Controller was very easy and simplified our deployment. We’re excited about continuing to test an environment with a minimized public footprint. Big thanks to Cloudflare and Stackpoint for the beta bits!
One question: Are there any special concerns wrt to scaling the Warp ingress controller pods? Or do we scale it just like any other service?
I discovered the source of my runaway CPU and non-functioning warp ingress. My ingress hostname had a subdomain component:
foo.default.acme.com, not foo.acme.com
Which I gather warp doesn’t like. I ran into the same issue running the Cloudflare-warp command line utility. If I trim off the sub domain, it seems to work fine.
Not sure if this qualifies as bug or not - but it might be good to document this.
Thanks for pointing out the issue! The certificate issued when you first log in only supports *acme.com and acme.com. We are working on providing better error messages.
I think if we parsed the certificate data on the client side (in the controller code) we could extract the SANs and check them against the requested hostname, and bail out with a good error message before we even request a tunnel. Obviously, “parse the certificate” is a real pain, but may be worth it.
I am finding some weird behavior - which is puzzling me. It seems like warp does not properly handle 302 redirects (or maybe it is my app?)
The app has a landing page on /, and an admin page on /admin/
If you go directly to /admin/ (note trailing /) - it works as expected.
The link on the landing page is to /admin (note - no trailing /) - which triggers the application to issue a 302 redirect to /admin/. It looks like the app sends the 302, but Cloudflare doesn’t send it back to the browser. The browser just throws up a CF error page complaining about a faulty dns name.
FWIW - The nginx ingress controller seems to handle this redirect just fine.
I also see this behaviour on another application. It sends a 302 - but the browser never sees it. But in this case, I can see CF request the 302 URL target - but the request never makes it back to the browser.
I tried fiddling with the ingress path (does it take wildcards? /* ) and the ingress.kubernetes.io/rewrite-target: / annotation. No substantial change.
Hints?
Hi Dani.
I followed the quick-start guide (https://warp.Cloudflare.com/docs/kubernetes/ as of 2017-12-15) and everything worked as magic, except for “Step Two” (the command copy-pasted from Chrome browser into MacOS terminal created an empty secret in the cluster).
But when I tried adding multiple rules, I got an error:
controller.go:140] Cannot create tunnel for ingress with multiple rules
Is there any way to support the configuration below?
rules:
- host: www.example.com
http:
paths:
- path: /api/
backend:
serviceName: example-api
servicePort: 80
- path: /
backend:
serviceName: website
servicePort: 80
- host: metabase.example.com
http:
paths:
- path: /
backend:
serviceName: metabase
servicePort: 80
Thanks,
Vladimir
The simple thing to do is to have multiple ingresses. Because there are two hostnames, and two services, the controller must create two tunnels. While it would be possible to manage the two tunnels associated with a single ingress, it seems simpler to me, architecturally, to have one-to-one relationships between service and ingress and tunnel and hostname.
Nate, got it, thanks. Sounds totally reasonable.
(and it would have helped me if I could read your hint about multiple ingresses or a link to docs right from the logs)
PS: I noticed that the controller pod consumes way too much CPU resources. Not sure if this is related to my prior experiments with multiple rules. I filed an issue here: https://github.com/Cloudflare/Cloudflare-warp-ingress/issues/2
I’m also seeing the high CPU usage as @vlad.yartsev. I don’t feel like we noticed this occurring until this past week.