Argo-ingress not working anymore. lookup cftunnel.com on 10.0.0.10:53: no such host;

shantanuthatte · August 12, 2018, 6:51pm

Hi,

I have argo-ingress deployed in my cluster from GitHub - cloudflare/cloudflare-ingress-controller: A Kubernetes ingress controller for Cloudflare's Argo Tunnels

I had cloned the repo and installed using:

helm install --name $RELEASE_NAME --namespace $NS --set rbac.install=$USE_RBAC --set secret.install=true,secret.domain=$DOMAIN,secret.certificate_b64=$CERT_B64 --set image.pullPolicy=Always ./chart --debug

Recently (past 4 days), I am seeing 503 issues:

503 Service Unavailable
The origin has been unregistered from Argo Tunnel

This was resolved by restarting the pod. Going through the logs of these pods, there wasn’t any error suggesting that the tunnel was closed. I could still see Validation ok for running default/httpbin/httpbin with 1 endpoint(s).

Today however, all my ingresses have stopped working. I get the usual log of ingress, creating tunnel and validation ok. But I don’t see the log message which states the PoP connected to (ex: connected to SIN). I just see:

controller.go:722] Starting tunnel to url httpbin.default:80
controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
controller.go:635] created tunnel for ingress httpbin, default/httpbin/httpbin

Running Cloudflared directly with --hello-world works and creates the appropriate CNAME records. I ssh’d into the pod and ran argot manually. Then I deleted all pods to a service, which is caught by argot as endpoint unavailable. When the pod is re-created by the replication-controller, I see the following events:

controller.go:307] Watching endpoint default/httpbin
controller.go:694] Endpoints not ready for tunnel default/httpbin/httpbin
controller.go:502] Error processing update:default/httpbin: at least one error occured handling update:default/httpbin: lookup cftunnel.com on 10.0.0.10:53: no such host
controller.go:694] Endpoints not ready for tunnel default/httpbin/httpbin
controller.go:307] Watching endpoint default/httpbin
controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
controller.go:722] Starting tunnel to url httpbin.default:80

To diagnose the lookup error, I ran the following commands from another pod in same namespace:

/ # dig +trace a cftunnel.com

; <<>> DiG 9.11.3 <<>> +trace a cftunnel.com
;; global options: +cmd
.                       241933  IN      NS      m.root-servers.net.
.                       241933  IN      NS      b.root-servers.net.
.                       241933  IN      NS      c.root-servers.net.
.                       241933  IN      NS      d.root-servers.net.
.                       241933  IN      NS      e.root-servers.net.
.                       241933  IN      NS      f.root-servers.net.
.                       241933  IN      NS      g.root-servers.net.
.                       241933  IN      NS      h.root-servers.net.
.                       241933  IN      NS      i.root-servers.net.
.                       241933  IN      NS      a.root-servers.net.
.                       241933  IN      NS      j.root-servers.net.
.                       241933  IN      NS      k.root-servers.net.
.                       241933  IN      NS      l.root-servers.net.
.                       241933  IN      RRSIG   NS 8 0 518400 20180825050000 20180812040000 41656 . AKpBAC+GLUffj3ssEoEkbd03Kcsq+yKvzaLIorw4kcwWeXGiD7zvECyb 74erZSoeA25J4W75bUyetwOEj+JVoTey5mPxQGyIR2t5sRKrHdKDJiSs BsW5gvayV/m+3BltYSQhUEihzbmEcj6JZLCAZxlH1C7KyXeOInDK5XYg epSMumair6RiMNaIm7zH74jFG5BiIjXo/oAprDiPP5oWqBMJNOgkdAvz LZNENPFweTEskKzXOsTP3V0MQxqxPcmTbe4G3WEAkrD7TiFJZLK/1nWZ NgFZ5IcTGo/QxgWEiycfyaM2sdqXHQ+JMptSrJvcfYWnPxM+Z7YfDQ6z koaD5A==
;; Received 525 bytes from 10.0.0.10#53(10.0.0.10) in 0 ms

com.                    172800  IN      NS      a.gtld-servers.net.
com.                    172800  IN      NS      b.gtld-servers.net.
com.                    172800  IN      NS      c.gtld-servers.net.
com.                    172800  IN      NS      d.gtld-servers.net.
com.                    172800  IN      NS      e.gtld-servers.net.
com.                    172800  IN      NS      f.gtld-servers.net.
com.                    172800  IN      NS      g.gtld-servers.net.
com.                    172800  IN      NS      h.gtld-servers.net.
com.                    172800  IN      NS      i.gtld-servers.net.
com.                    172800  IN      NS      j.gtld-servers.net.
com.                    172800  IN      NS      k.gtld-servers.net.
com.                    172800  IN      NS      l.gtld-servers.net.
com.                    172800  IN      NS      m.gtld-servers.net.
com.                    86400   IN      DS      30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CF C41A5766
com.                    86400   IN      RRSIG   DS 8 1 86400 20180825050000 20180812040000 41656 . rTOR/bcUhIjlLufuHmwodcGHCV1T3McqK08tTtHgBwmGUS/CAxD7LE0l R8RRsuhu1F3En6MIbz69/RLWURm8S69QIPkrXLMXko+k5bW2IWJatPe3 IxeswJl9gN2/oKsHD4UnpLJ+amLSUH0gZ44yFQpiyelRpg+GYHk2L6r2 4yjBi2+Gz0wVTmZZeY3GeluTstIAu/35LPhiLEiwJwf7WfhAQWka+noE 60I+5Ne21l74Gffkn5S5UmEOc+eihtj1v/RADyO2F6p/KuazVLwjbD/b RDlqG93y+ObsmxJwnRLg2eEt4+hnl7MLRXMyYsboShIeH0ox+5r+nW23 0gLkwA==
;; Received 1172 bytes from 193.0.14.129#53(k.root-servers.net) in 128 ms

cftunnel.com.           172800  IN      NS      kevin.ns.Cloudflare.com.
cftunnel.com.           172800  IN      NS      marjory.ns.Cloudflare.com.
cftunnel.com.           86400   IN      DS      2371 13 2 E628814E43E924733990D4B7DA1B2E3E42107ECB754AEDAA01B97D20 85648728
cftunnel.com.           86400   IN      RRSIG   DS 8 2 86400 20180818043630 20180811032630 46475 com. JRdrMEBKm2CXW8tev8FY8j6YrzA5M9eJQD6LIKJU826OOXL5McJ7yTsD 0R4jMg89xTLd7KCmHYoHkDNvONxwD9nNOhZvQ2O1BSzvuO5H89ADfdth MAv/jqeDEG1hmkIeXww0DzyLmoEbEfN6qAjB7nEAA215wjNmGR2yDik2 cE8=
;; Received 396 bytes from 192.26.92.30#53(c.gtld-servers.net) in 270 ms

cftunnel.com.           30      IN      SOA     kevin.ns.Cloudflare.com. dns.Cloudflare.com. 2028576619 10000 2400 604800 30
cftunnel.com.           30      IN      NSEC    \000.cftunnel.com. NS SOA HINFO MX TXT AAAA LOC SRV CERT SSHFP RRSIG NSEC DNSKEY TLSA HIP CDS CDNSKEY OPENPGPKEY SPF
cftunnel.com.           30      IN      RRSIG   SOA 13 2 30 20180813194652 20180811174652 35273 cftunnel.com. l5det6ze+36Ek8me415LC5cIK6j1GU4eVAoQmqaLk6NBEOzmZiXBfHZW x6isewP2UWYQ0QKUzKTb5YJzXTOw3w==
cftunnel.com.           30      IN      RRSIG   NSEC 13 2 30 20180813194652 20180811174652 35273 cftunnel.com. wGvw362LmbifiPDprnMsLx//ou+BoWxWK9VVtLTky6Uov87Xm0+L0YRu r8zcXHaXCu0mlCiMrEJy07fBeRsgEw==
;; Received 360 bytes from 173.245.58.193#53(marjory.ns.Cloudflare.com) in 64 ms

/ # dig a cftunnel.com

; <<>> DiG 9.11.3 <<>> a cftunnel.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44398
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cftunnel.com.                  IN      A

;; AUTHORITY SECTION:
cftunnel.com.           29      IN      SOA     kevin.ns.Cloudflare.com. dns.Cloudflare.com. 2028576619 10000 2400 604800 30

;; Query time: 69 msec
;; SERVER: 10.0.0.10#53(10.0.0.10)
;; WHEN: Sun Aug 12 18:47:26 UTC 2018
;; MSG SIZE  rcvd: 101

/ # dig cftunnel.com ANY

; <<>> DiG 9.11.3 <<>> cftunnel.com ANY
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8216
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;cftunnel.com.                  IN      ANY

;; ANSWER SECTION:
cftunnel.com.           3788    IN      HINFO   "ANY obsoleted" "See draft-ietf-dnsop-refuse-any"
cftunnel.com.           3788    IN      RRSIG   HINFO 13 2 3789 20180813194828 20180811174828 35273 cftunnel.com. IQ58x0PakpDhBFlgtukJzI37ti6kCgE81B0Lc3P1RNX4ABcubbnLylAq BGhaMuSxh8t3AV/+IkJOFHkG4WTsCA==

;; Query time: 69 msec
;; SERVER: 10.0.0.10#53(10.0.0.10)
;; WHEN: Sun Aug 12 18:48:28 UTC 2018
;; MSG SIZE  rcvd: 207

Looks like the lookup of an A record to cftunnel.com is not working. Or some other issue with argot.

Any help with this issue is greatly appreciated.

Thanks,
Shantanu

floran · August 14, 2018, 7:55am

We are having a similar the same issue, the origin is not created in the traffic page, no matter how hard I try.
Quick help would be really appreciated

barry1 · August 14, 2018, 7:33pm

I got the same and due to tunnels falling over a lot actually have a script checking and restarting the argo pod if necessary. I found though recently that something had changed so I upgraded the container image to

gcr.io/stackpoint-public/argot:20180812-c

with the hash 7a55aedbf0b1

which did work but today subsequently disappeared from the repo!

I was able to retrieve a version of this working container image from one of the hosts however and retagged and pushed to public docker hub as

barrymac/argobackup

It got my tunnels back up and running but we suffered some down time while I was frantically retrieving the image.

Latest stable one 0.5.1 is not working for me.

ryan · August 14, 2018, 8:53pm

Are you all still experiencing issues, or is it resolved?

shantanuthatte · August 14, 2018, 10:05pm

I had to remove argo-ingress and add a static IP with a LoadBalancer to our production backend. I haven’t yet re-enabled it on our staging. I’ll try and let you know by tomorrow. Is 0.5.1 the image tag we should be using?

Also, I’m interested in knowing the technicalities of the issue (if you guys found something).

ryan · August 14, 2018, 10:08pm

I need to make sure the issue is still happening and is reproducable before I ask engineers to jump off of other tasks. So let me know the latest status and reproduction steps and I’ll get it filed right away.

shantanuthatte · August 15, 2018, 12:52pm

Hi,

I checked, the latest 0.5.1 image is still experiencing the same issue.

Steps to reproduce:

Install the Helm chart following the GitHub repo README.

helm install --name $RELEASE_NAME --namespace $NS  --set rbac.install=$USE_RBAC    --set secret.install=true,secret.domain=$DOMAIN,secret.certificate_b64=$CERT_B64 ./chart

Install httpbin as in README.
Made sure that the ingress class is argo-tunnel
Expect to see CNAME record changed to the tunnel and a tunnel being created.

Logs:

argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.043418 1 main.go:67] Starting Controller
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.044388 1 controller.go:318] Starting ArgoController
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.044795 1 reflector.go:202] Starting reflector *v1.Service (1m0s) from github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.044846 1 reflector.go:240] Listing and watching *v1.Service from github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.044814 1 reflector.go:202] Starting reflector *v1.Endpoints (1m0s) from github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.044964 1 reflector.go:240] Listing and watching *v1.Endpoints from github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.045611 1 reflector.go:202] Starting reflector *v1beta1.Ingress (1m0s) from github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.045628 1 reflector.go:240] Listing and watching *v1beta1.Ingress from github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.061661 1 round_trippers.go:436] GET https://10.0.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0 200 OK in 16 milliseconds
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.143592 1 round_trippers.go:436] GET https://10.0.0.1:443/apis/extensions/v1beta1/ingresses?limit=500&resourceVersion=0 200 OK in 97 milliseconds
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.147028 1 round_trippers.go:436] GET https://10.0.0.1:443/api/v1/services?limit=500&resourceVersion=0 200 OK in 101 milliseconds
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.148549 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.148723 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.148829 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.149135 1 round_trippers.go:436] GET https://10.0.0.1:443/apis/extensions/v1beta1/ingresses?resourceVersion=5346462&timeoutSeconds=481&watch=true 200 OK in 2 milliseconds
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.150560 1 round_trippers.go:436] GET https://10.0.0.1:443/api/v1/endpoints?resourceVersion=5511974&timeoutSeconds=582&watch=true 200 OK in 2 milliseconds
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.244559 1 shared_informer.go:122] caches populated
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.245478 1 round_trippers.go:436] GET https://10.0.0.1:443/api/v1/services?resourceVersion=5416367&timeoutSeconds=499&watch=true 200 OK in 2 milliseconds
argo-ingress    Aug 15, 2018, 6:13:54 PM    I0815 12:43:54.344731 1 shared_informer.go:122] caches populated
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.013918 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.013956 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.013986 1 controller.go:609] creating tunnel for ingress httpbin, default/httpbin/httpbin
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.024492 1 round_trippers.go:436] GET https://10.0.0.1:443/api/v1/namespaces/argo/secrets/Cloudflared-cert 200 OK in 10 milliseconds
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.043823 1 controller.go:635] created tunnel for ingress httpbin, default/httpbin/httpbin
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.043861 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.043928 1 controller.go:722] Starting tunnel to url httpbin.default:80
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.047694 1 round_trippers.go:436] GET https://10.0.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/httpbin 200 OK in 3 milliseconds
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.143512 1 round_trippers.go:436] PUT https://10.0.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/httpbin/status 200 OK in 95 milliseconds
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.143951 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:14:47 PM    I0815 12:44:47.144044 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.147184 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322: forcing resync
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.147351 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.147384 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.147430 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.147451 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.147472 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.147511 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.148661 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321: forcing resync
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.148928 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.149088 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.243475 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320: forcing resync
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.243708 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:14:54 PM    I0815 12:44:54.243766 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.147480 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322: forcing resync
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.147573 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.147615 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.147656 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.147684 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.147701 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.147713 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.149041 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321: forcing resync
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.149250 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.149278 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.243876 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320: forcing resync
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.244004 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:15:54 PM    I0815 12:45:54.244101 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.147701 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322: forcing resync
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.147954 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.147985 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.147998 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.148019 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.148051 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.148087 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.149798 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321: forcing resync
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.150040 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.150229 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.244204 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320: forcing resync
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.244295 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:16:54 PM    I0815 12:46:54.244339 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.147963 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322: forcing resync
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.148252 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.148276 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.148302 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.148318 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.148339 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.148350 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.150593 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321: forcing resync
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.150719 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.150760 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.244446 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320: forcing resync
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.244587 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:17:54 PM    I0815 12:47:54.244637 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.148381 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322: forcing resync
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.148509 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.148516 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.148523 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.148527 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.148532 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.148538 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.151047 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321: forcing resync
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.151149 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.151192 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.244742 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320: forcing resync
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.244831 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:18:54 PM    I0815 12:48:54.244862 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.148633 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322: forcing resync
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.148738 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.148756 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.148797 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.148820 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.148858 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.148882 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.151733 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321: forcing resync
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.151944 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.152047 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.245025 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320: forcing resync
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.245134 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:19:54 PM    I0815 12:49:54.245179 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)

Now delete httpbin pod and wait for replication controller to start a new one.

argo-ingress    Aug 15, 2018, 6:25:54 PM    I0815 12:55:54.246987 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:25:54 PM    I0815 12:55:54.247128 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:26:31 PM    I0815 12:56:31.985247 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:26:31 PM    I0815 12:56:31.985296 1 controller.go:694] Endpoints not ready for tunnel default/httpbin/httpbin
argo-ingress    Aug 15, 2018, 6:26:31 PM    I0815 12:56:31.985314 1 controller.go:502] Error processing update:default/httpbin: at least one error occured handling update:default/httpbin: lookup cftunnel.com on 10.0.0.10:53: no such host
argo-ingress    Aug 15, 2018, 6:26:31 PM    I0815 12:56:31.990483 1 controller.go:694] Endpoints not ready for tunnel default/httpbin/httpbin
argo-ingress    Aug 15, 2018, 6:26:38 PM    I0815 12:56:38.858972 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:26:38 PM    I0815 12:56:38.859274 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:26:38 PM    I0815 12:56:38.859435 1 controller.go:722] Starting tunnel to url httpbin.default:80
argo-ingress    Aug 15, 2018, 6:26:38 PM    I0815 12:56:38.867109 1 round_trippers.go:436] GET https://10.0.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/httpbin 200 OK in 7 milliseconds
argo-ingress    Aug 15, 2018, 6:26:38 PM    I0815 12:56:38.871482 1 round_trippers.go:436] PUT https://10.0.0.1:443/apis/extensions/v1beta1/namespaces/default/ingresses/httpbin/status 200 OK in 3 milliseconds
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243321 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:322: forcing resync
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243385 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:321: forcing resync
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243402 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243408 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243417 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243423 1 controller.go:185] Annotation kubernetes.io/ingress.class=nginx
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243431 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243439 1 controller.go:185] Annotation kubernetes.io/ingress.class=argo-tunnel
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243481 1 controller.go:307] Watching endpoint default/httpbin
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.243534 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.247067 1 reflector.go:286] github.com/Cloudflare/Cloudflare-ingress-controller/pkg/controller/controller.go:320: forcing resync
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.247181 1 controller.go:246] Watching service default/httpbin
argo-ingress    Aug 15, 2018, 6:26:54 PM    I0815 12:56:54.247218 1 controller.go:698] Validation ok for running default/httpbin/httpbin with 1 endpoint(s)

Also on a side-note, any way we can turn down the logging a notch via env so as not to have our log aggregator flooded?

zack · August 15, 2018, 4:55pm

Sorry for the trouble, I’m sure this has been very frustrating. We’re working to get a new released pushed today which should address this issue.

shantanuthatte · August 15, 2018, 5:11pm

Thanks a lot.

UPDATE:
If you have a docker image tag that I can help you test, let me know.

shantanuthatte · August 15, 2018, 5:17pm

Would the new release support multiple hosts/paths in the ingress definition? Or its something that would get incorporated at a later stage?

nate · August 15, 2018, 5:46pm

@shantanuthatte If you want to try a new build, there’s gcr.io/stackpoint-public/argot:ecf5bf3 which is close to release. I am not sure whether the Cloudflare communication problems are resolved there, but the build has a lot of dependency updates.

The log level is set in the kubernetes deployment spec, you’ll see the command entry has a --v=6 setting. You can push that down to 2 or 1 or 0 and remove most of the log messages. You can change it after installation with kubectl edit ... or before installation by editing ./chart/templates/deployment.yaml. It should be (will be) a setting in values.yaml

shantanuthatte · August 15, 2018, 6:13pm

Thanks, I updated the pod. Looks like there is an error a service account not having permission.

I0815 18:10:25.173194 1 controller.go:501] Error processing add:default/httpbin/httpbin: secrets is forbidden: User "system:serviceaccount:argo:Cloudflare-argo" cannot list secrets in the namespace "argo": Unknown user "system:serviceaccount:argo:Cloudflare-argo"

Seems this might not be a drop-in replacement.

I can still see the serviceaccount

apiVersion: v1
  kind: ServiceAccount
  metadata:
    creationTimestamp: 2018-08-15T12:43:43Z
    name: Cloudflare-argo
    namespace: argo
    resourceVersion: "5511938"
    selfLink: /api/v1/namespaces/argo/serviceaccounts/Cloudflare-argo
    uid: d837a953-a088-11e8-8d44-42010aa00fce
  secrets:
  - name: Cloudflare-argo-token-wjb4b

UPDATE:
I tried deleting the release and installing the chart again, with the same result.

nate · August 15, 2018, 6:53pm

Sorry, I changed some of the accesses and permissions for more security It’s configured correctly in the helm chart, but you can make yours work by editing a clusterrole, kubectl edit clusterrole Cloudflare-argo, and adding a new verb list, i.e.

- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - list
  - get

(that particular change is less secure, but in the helm chart it becomes a namespaced role etc etc)

shantanuthatte · August 16, 2018, 9:26am

Works with the new release (0.5.2)! Thanks a lot!