Argo Tunnel 502 error

I`m testing Argo tunnel and when I use service http://localhost:80 everything works fine, however when I sue service: https://localhost:443 it does not, I know Argo tunnel is already SSL but still want to test.

Cloudflared version ( When I run in from cli it says it is old but this is latest I could download from dev anyway )

[email protected]:~# cloudflared --v
cloudflared version 2021.5.9 (built 2021-05-21-1541 UTC)
[email protected]:~#

Cloudflared Config:

[email protected]:~# cat /etc/cloudflared/config.yml
tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json


ingress:
 - hostname: www.mpabian.co.uk
   service: https://localhost:443
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404

When I test Rules it shows correct service:

[email protected]:~# cloudflared tunnel --config /etc/cloudflared/config.yml ingress rule https://www.mpabian.co.uk
Using rules from /etc/cloudflared/config.yml
Matched rule #1
        hostname: www.mpabian.co.uk
        service: https://localhost:443
[email protected]:~#

When I curl it with localhost it works fine:

[email protected]:~# curl https://localhost:443 -H "host:www.mpabian.co.uk" -Ivk
*   Trying ::1...
* TCP_NODELAY set
* Expire in 149998 ms for 3 (transfer 0x55e9370cdf90)
* Expire in 200 ms for 4 (transfer 0x55e9370cdf90)
* Connected to localhost (::1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: O=CloudFlare, Inc.; OU=CloudFlare Origin CA; CN=CloudFlare Origin Certificate
*  start date: Jun 15 13:37:00 2021 GMT
*  expire date: Jun 11 13:37:00 2036 GMT
*  issuer: C=US; O=CloudFlare, Inc.; OU=CloudFlare Origin SSL Certificate Authority; L=San Francisco; ST=California
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> HEAD / HTTP/1.1
> Host:www.mpabian.co.uk
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 30 Jun 2021 13:44:13 GMT
Date: Wed, 30 Jun 2021 13:44:13 GMT
< Server: Apache/2.4.38 (Debian)
Server: Apache/2.4.38 (Debian)
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8

<
* Connection #0 to host localhost left intact

Only way I was able to make it over SSL through tunnel it is with this config, also I had to add no-tls-verify: true despite i`m hosting ORGIN Certificate from Cloudlfare

[email protected]:~# cat /etc/cloudflared/config.yml
tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
hostname: www.mpabian.co.uk
url: https://localhost:443
no-tls-verify: true


For any subdomain using just http://localhost:random-port this seems to be working fine

You’ll need to either use the no-tls-verify: true setting in your config or install the Cloudflare Root certificate on your system.

https://developers.cloudflare.com/cloudflare-one/connections/connect-devices/warp/install-cloudflare-cert

Hello Garret,

Thank yo for response,

I`m afraid bellow config did not work.

[email protected]:~# cat /etc/cloudflared/config.yml
tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
#hostname: www.mpabian.co.uk
#url: https://localhost:443
no-tls-verify: true

ingress:
 - hostname: www.mpabian.co.uk
   service: https://localhost:443
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404

I have tried no-tls-verify: true under hostname itself as well and of course I have restarted process.

To be honest Im not sure how installing cloudflare CA cert would help I do not think solution for installing Cert would work either as Cloudflare is authenticating server not server authenticating cloudflare (client) unless im missing how argo tunnel works under the hood but from technical perspective does not make much sense unless you would like to do Man in the Middle where CF would sign certificates for sites I access but that completely different topic.

Regards,
Miroslaw Pabian

Where did you download it from? It is indeed not the latest version.

As for the 502 errors, I looked up your tunnel 46e47211-d901-4306-a596-c21df1b85c01 and I could not find any DNS record pointing to it.

Did you do these steps? https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/routing-to-tunnel/dns

Hello Nuno,

Thank you for Response!

i have downloaded it from:

https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/installation ->  https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-amd64.deb

In regards to tunnel setup is little bit complex maybe that is why you do not see records for it.

so my domain www.mpabian.co.uk is on one cloudflare account, i`m proxing to other cloud account (enterprise plan ) with custom hostnames then i point to LB with in that account, after that I use LB rules to set correct pools based on hostname. So argo tunnel is in one of pools.

I have moved service to port 80 and you can see it works now

[email protected]:~# cat /etc/cloudflared/config.yml
tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
#hostname: www.mpabian.co.uk
#url: https://localhost:443
no-tls-verify: true

ingress:
 - hostname: www.mpabian.co.uk
   service: http://localhost:80
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404
[email protected]:~#

to confirm that you can see on the page header added CLOUDFLARE_CONNECTING_IP and ORGIN_IP both are ::1 what shows it is going through tunnel

[email protected]:~# cloudflared tunnel list
You can obtain more detailed information for each tunnel with `cloudflared tunnel info <name/uuid>`
ID                                   NAME        CREATED              CONNECTIONS
46e47211-d901-4306-a596-c21df1b85c01 argo-ovh    2021-06-04T09:12:17Z 2xKBP, 2xWAW
3972e13d-449e-4964-b111-59281b73d1d8 argo-ovh2   2021-06-09T15:48:36Z 2xKBP, 2xWAW
dd137af4-e24c-455b-95ea-4ea1fe7a3096 ivo-cf-argo 2021-06-30T04:21:00Z 2xMEL, 2xSYD
2021-07-01T08:36:13Z WRN Your version 2021.5.9 is outdated. We recommend upgrading it to 2021.6.0
[email protected]:~#

Regards,
Miroslaw Pabian

Thanks for clarifying Miroslaw.

When you set it up with origin being https in cloudflared config, and send a request to it, is there any error on cloudflared tunnel logs?

Hey Nuno

yes I should think of that

{“level”:“error”,“error”:“Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: x509: certificate is valid for *.cflab.xxx.net, cflab.xxx.net, not localhost”,“cfRay”:“667e7b5c2c0327b4-KBP”,“ingressRule”:“0”,“originService”:“https://localhost:443”,“time”:“2021-07-01T09:07:03Z”}

seems like cloudflared expect cert for “localhost” because that how it is defined in the config :confused: strange
anyway I have disabled verification in any possible way

[email protected]:/var/log# cat /etc/cloudflared/config.yml
tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
logfile: /var/log/cloudflared
loglevel: trace
transport-loglevel: trace
no-tls-verify: true

ingress:
 - hostname: www.mpabian.co.uk
   noTLSVerify: true
   service: https://localhost:443
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404
[email protected]:/var/log#

But still same issue so not sure if i`m putting options in wrong place

Regards,
Miroslaw Pabian

I suppose you can use the originServerName property under the ingress rule to define a name that will match the SSL certificate presented by your origin (see https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress#originServerName) and it should then work.

Yes seems like none of options have any effect, this one as well ( I have restarted Tunnel )


tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
logfile: /var/log/cloudflared
loglevel: error
transport-loglevel: error

ingress:
 - hostname: www.mpabian.co.uk
   originServerName: cflab.XXX.net
   service: https://localhost:443
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404
[email protected]:~#


{"level":"error","error":"Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: x509: certificate is valid for *.cflab.XXX.net, cflab.XXX.net, not localhost","cfRay":"667eb9737ca34db2-WAW","ingressRule":"0","originService":"https://localhost:443","time":"2021-07-01T09:49:27Z"}

Regards,
Miroslaw Pabian

Now that’s odd, because I reproduced your setup and with originServerName cloudflared accepted the SSL certificate that’s presented for that origin name.

I can’t tell what else may be different, but from my perspective, that should work.

Hello Nuno

Agreed It should, not sure if I update cloudflared will it help at all but where can I download latest version if not from dev site as it is not there? ( I do not want to compile ) deb file would be fine

Regards,
Miroslaw Pabian

I don’t think anything related to this has changed recently.

In any case, the latest version is always available at Releases · cloudflare/cloudflared · GitHub and Download Cloudflared
You can also just run cloudflared update and the binary will update itself.

Thank that is updated now, I will do some more testing and see what is happening.
I have simplified configuration to not proxy as well so you should be able to see my argo CNAMED for domain.

How your config that works looks like ?

Regards,
Miroslaw Pabian

Pretty much the same as yours.

tunnel: ...
credentials-file: ...

ingress:
 - hostname: ...
   originServerName: ...
   caPool: ...
   service: https://localhost:443
 - service: http_status:404

The only difference is that I had to pass a caPool to provide the CA pem to cloudflared for it to be able to validate the SSL certificate presented by my origin. But that’s just because the certificate my origin is presenting cannot be validated by my system otherwise, since I do not have that CA pem installed in my system where cloudflared is running.
I don’t think this applies to you based on what you said above.

Thank you,

This really going to drive me crazy hehe :slight_smile:

As per documentation: caPool Path to the CA for the certificate of your origin. This option should be used only if your certificate is not signed by Cloudflare.

My certificate is signed by CF as I use ORGIN certificate, so in theory I would not have to do caPool but I have done it

tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
logfile: /var/log/cloudflared
loglevel: error
transport-loglevel: error

ingress:
 - hostname: mpabian-cache.cflab.XXX.net
   service: https://localhost:443
 - hostname: www.mpabian.co.uk
   originServerName: cflab.XXX.net
   caPool: /etc/cloudflared/CloudFlare_Origin_SSL_Certificate_Authority.pem
   service: https://localhost:443
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404


I found this certificate to be for orgin https://developers.cloudflare.com/ssl/e2b9968022bf23b071d95229b5678452/origin_ca_rsa_root.pem

I still have error


{"level":"error","error":"Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: x509: certificate is valid for *.cflab.XXX.net, cflab.XXX.net, not localhost","cfRay":"667ff09d2cd4542d-WAW","ingressRule":"1","originService":"https://localhost:443","time":"2021-07-01T13:21:52Z"}

Only way to go around that was to use following config, so basically I pointed domain that I have cert for from CF in hostfile to be 127.0.0.1 and then refer that in cloudflared config

[email protected]:/etc/cloudflared# cat  config.yml
tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
logfile: /var/log/cloudflared
loglevel: error
transport-loglevel: error

ingress:
 - hostname: mpabian-cache.cflab.XXX.net
   service: https://localhost:443
 - hostname: www.mpabian.co.uk
   originServerName: cflab.XXX.net
   caPool: /etc/cloudflared/CloudFlare_Origin_SSL_Certificate_Authority.pem
   service: https://cflab.XXX.net:443
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404


[email protected]:/etc/cloudflared# cat /etc/hosts
127.0.0.1       localhost

::1     localhost       ip6-localhost   ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters


127.0.0.1       cflab.XXX.net


I want to scream Help help help :smiley:

Hello Nunu,

What appears if you test your config ?

When I test it shows

[email protected]:/etc/cloudflared# cloudflared tunnel --config /etc/cloudflared/config.yml ingress rule https://www.mpabian.co.uk
Using rules from /etc/cloudflared/config.yml
Matched rule #2
        hostname: www.mpabian.co.uk
        service: https://localhost:443

but I have this now:

tunnel: 46e47211-d901-4306-a596-c21df1b85c01
credentials-file: /root/.cloudflared/46e47211-d901-4306-a596-c21df1b85c01.json
logfile: /var/log/cloudflared
loglevel: error
transport-loglevel: error

ingress:
 - hostname: mpabian-cache.cflab.xxx.net
   service: https://localhost:443
 - hostname: www.mpabian.co.uk
   originServerName: cflab.xxx.net
   noTLSVerify: true
   service: https://localhost:443
 - hostname: www.mpabian.uk
   service: http://localhost:80
 - service: http_status:404
[email protected]:/etc/cloudflared#

I`m not sure if I should expect to show

  hostname: www.mpabian.co.uk
   originServerName: cflab.xxx.net
   noTLSVerify: true
   service: https://localhost:443

Regards,
Miroslaw Pabian