Troubleshooting Argo Tunnel

I’m hosting a NextCloud instance behind a NGINX reverse proxy.

Everything works (ie: I’m able to get to the login page of nextcloud that’s hosted internally) when Cloudflare DNS CNAME record (nextcloud) is setup to be Proxied through my domain’s A record; however, when I change the CNAME to go through the Argo tunnel via the tunnel-id, I get a HTTP 502/Bad Gateway error. BTW, my Argo Tunnel is setup and working as I’m hosting other internal services via the tunnel.

To summarize, when accessing internal service like NextCloud via Cloudflare Proxy that goes through my reverse proxy, things work; however, when going through the Argo Tunnel (ie: by passing the nginx reverse proxy), I get a Bad Gateway error.

Any ideas why this could be happening?

On your machine running the cloudflared instance if you do something like curl -Ikv https://www.example.com -resolve www.example.com:443:my.internal.ip.address does that work?

If you do a dig for whatever hostname you’ve specified as the origin for that host on the machine running cloudflared does it resolve to the proper host/IP?

Thanks so much for a quick response!

I wanted to confirm the command syntax:

curl -Ikv https://nextcloud.mydomain.com -resolve nextcloud.mydomain.com:443:192.168.2.20

?

Doing a ‘dig nextcloud.mydoman.com’ does return two A records for Cloudflare Proxy:
image

I doubt that this is a DNS resolution issue since I can access the internal service via the Cloudflare Proxy. The issue might be more Tunnel related.

curl -Ikv https://nextcloud.mydomain.com --resolve nextcloud.mydomain.com:443:192.168.2.20

needs an extra dash… my apologies… it’s been a long week. :slight_smile:

1 Like

TGIF !! :slight_smile:

Command Run:

curl -Ikv https://nextcloud.mydomain.com -resolve nextcloud.mydomain.com:444:192.168.2.20

Please note that the target origin port is running https on 444 (instead of the default 443).

Here’s the output with the exact domain & IPs masked and/or changed:

`* Added nextcloud.mydomain.com:444:192.168.2.20 to DNS cache

  • Trying 2606:4700:XXXX:XXXX:XXXX:443…
  • Connected to nextcloud.mydomain.com (2606:4700:XXXX:XXXX:XXXX) port 443 (#0) ALPN, offering h2 ALPN, offering http/1.1 successfully set certificate verify locations:
  • CAfile: none
  • CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
  • start date: Jan 21 00:00:00 2021 GMT
  • expire date: Jan 20 23:59:59 2022 GMT
  • issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x47c540)

HEAD / HTTP/2
Host: nextcloud.mydomain.com
user-agent: curl/7.73.0
accept: /

<

Is your yml file pointing to something like https://192.168.2.20:444 ?

It’s pointing to https://localhost:444

What happens if you change it to this?

Hi guys,

I have a very similar situation when trying to add services (dockers) in my UnRAID server at home.

I’m using NginxProxyManager docker, and this is how it looks:

I created my origin certificate and using it for all the proxy hosts.

My config.yaml looks like this:

tunnel: 02c0092f-xxxx-xxx-xxxx-efde75ff8964
credentials-file: /home/nonroot/.cloudflared/02c0092f-xxxx-xxx-xxxx-efde75ff8964.json

# NOTE: You should only have one ingress tag, so if you uncomment one block comment the others

# forward all traffic to Reverse Proxy w/ SSL
ingress:
  - service: https://192.168.0.10:18443
    originRequest:
      originServerName: nc.my-domain.com

#forward all traffic to Reverse Proxy w/ SSL and no TLS Verify
#ingress:
#  - service: https://REVERSEPROXYIP:PORT
#    originRequest:
#      noTLSVerify: true

# forward all traffic to reverse proxy over http
#ingress:
#  - service: http://REVERSEPROXYIP:PORT

The reason I am using a subdomain as my origin server is because it does not work with the root domain.

According to Cloudflare Tunnel - Cloudflare Tunnel it should work with any subdomain like this, however, when I try to use any subdomain other than nc (for NextCloud) I always get an error 502 from Cloudflare.

If I run:

curl -Ikv https://nc.my-domain.com --resolve nc.my-domain.com:444:192.168.0.10

I get these results:

* Added nc.my-domain.com:444:192.168.0.10 to DNS cache
*   Trying 104.21.57.60:443...
* Connected to nc.my-domain.com (104.21.57.60) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: none
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Feb  3 00:00:00 2022 GMT
*  expire date: Feb  2 23:59:59 2023 GMT
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x494b00)
> HEAD / HTTP/2
> Host: nc.my-domain.com
> user-agent: curl/7.79.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 302 
HTTP/2 302 
< date: Fri, 04 Feb 2022 17:04:24 GMT
date: Fri, 04 Feb 2022 17:04:24 GMT
< content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8
< location: https://nc.my-domain.com/login
location: https://nc.my-domain.com/login
< cache-control: no-store, no-cache, must-revalidate
cache-control: no-store, no-cache, must-revalidate
< content-security-policy: default-src 'self'; script-src 'self' 'nonce-YjdDa1NKQWp0R2VLaEdGYjBZdzUyVnFnWjdtM0pYZHhVd3NxNDkxOVFaTT06SU9icUNxWlUxUlBEL1JBNW9QeGVyalhDTThQRUVBVVVCVmxhMitVVEp0ND0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';
content-security-policy: default-src 'self'; script-src 'self' 'nonce-YjdDa1NKQWp0R2VLaEdGYjBZdzUyVnFnWjdtM0pYZHhVd3NxNDkxOVFaTT06SU9icUNxWlUxUlBEL1JBNW9QeGVyalhDTThQRUVBVVVCVmxhMitVVEp0ND0='; style-src 'self' 'unsafe-inline'; frame-src *; img-src * data: blob:; font-src 'self' data:; media-src *; connect-src *; object-src 'none'; base-uri 'self';
< expires: Thu, 19 Nov 1981 08:52:00 GMT
expires: Thu, 19 Nov 1981 08:52:00 GMT
< pragma: no-cache
pragma: no-cache
< referrer-policy: no-referrer
referrer-policy: no-referrer
< set-cookie: oc_sessionPassphrase=FA01vnzm1ZQmr25UP1C%2BSnT9gUFifKElOdF3Qui8oLbNMCftXndK488usHSKrge3b0nfZsd4MR8LWzRoBfLkdfA1kHHCCzlxzx6ofSr8jqF%2FuBZRt8kIgifOLU4djQfc; path=/; secure; HttpOnly; SameSite=Lax
set-cookie: oc_sessionPassphrase=FA01vnzm1ZQmr25UP1C%2BSnT9gUFifKElOdF3Qui8oLbNMCftXndK488usHSKrge3b0nfZsd4MR8LWzRoBfLkdfA1kHHCCzlxzx6ofSr8jqF%2FuBZRt8kIgifOLU4djQfc; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: ocbbybzf14ew=mktf5gihqiluihdif096q7p28i; path=/; secure; HttpOnly; SameSite=Lax
set-cookie: ocbbybzf14ew=mktf5gihqiluihdif096q7p28i; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
set-cookie: __Host-nc_sameSiteCookielax=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
set-cookie: __Host-nc_sameSiteCookiestrict=true; path=/; httponly;secure; expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< strict-transport-security: max-age=63072000; preload
strict-transport-security: max-age=63072000; preload
< x-content-type-options: nosniff
x-content-type-options: nosniff
< x-download-options: noopen
x-download-options: noopen
< x-frame-options: SAMEORIGIN
x-frame-options: SAMEORIGIN
< x-permitted-cross-domain-policies: none
x-permitted-cross-domain-policies: none
< x-robots-tag: none
x-robots-tag: none
< x-served-by: nc.my-domain.com
x-served-by: nc.my-domain.com
< x-xss-protection: 1; mode=block
x-xss-protection: 1; mode=block
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=t%2B8SLbS4qy3ooSCbyvGbamNd47Nd%2FBNbS9mAsEChCEny5SA3XkYFpAuEXXjE2Cctf5n0hkVGnUFuE81NoDZ1vUPZbkTYrWi6IiEHo18WdjZ%2B6qv2YQqrKYDT3sx5FSMDXkbUwPk%3D"}],"group":"cf-nel","max_age":604800}
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=t%2B8SLbS4qy3ooSCbyvGbamNd47Nd%2FBNbS9mAsEChCEny5SA3XkYFpAuEXXjE2Cctf5n0hkVGnUFuE81NoDZ1vUPZbkTYrWi6IiEHo18WdjZ%2B6qv2YQqrKYDT3sx5FSMDXkbUwPk%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
server: cloudflare
< cf-ray: 6d857a4b5bc7e116-IAD
cf-ray: 6d857a4b5bc7e116-IAD
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

< 
* Connection #0 to host nc.my-domain.com left intact

And it works perfectly fine, but if I try with (“bw” is for my vaultwarden docker):

curl -Ikv https://bw.my-domain.com --resolve bw.my-domain.com:4743:192.168.0.10

The results are these:

* Added bw.my-domain.com:4743:192.168.0.10 to DNS cache
*   Trying 172.67.159.228:443...
* Connected to bw.my-domain.com (172.67.159.228) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: none
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Feb  3 00:00:00 2022 GMT
*  expire date: Feb  2 23:59:59 2023 GMT
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x494b00)
> HEAD / HTTP/2
> Host: bw.my-domain.com
> user-agent: curl/7.79.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 502 
HTTP/2 502 
< date: Fri, 04 Feb 2022 17:02:59 GMT
date: Fri, 04 Feb 2022 17:02:59 GMT
< content-type: text/html
content-type: text/html
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=%2Bialhy6Yk2wOfqIzpZhB5hARq8HLYARVC6%2FuZ7yM5ZgTQHuYRA3%2B8AGRfK9K5Y1qTKVC9Ttj46iqvrw6obnzgy8803DNyGn1ML4Yb%2FnrNaLodrilxHWnS1ep3eY9tsSRnVja42o%3D"}],"group":"cf-nel","max_age":604800}
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=%2Bialhy6Yk2wOfqIzpZhB5hARq8HLYARVC6%2FuZ7yM5ZgTQHuYRA3%2B8AGRfK9K5Y1qTKVC9Ttj46iqvrw6obnzgy8803DNyGn1ML4Yb%2FnrNaLodrilxHWnS1ep3eY9tsSRnVja42o%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
server: cloudflare
< cf-ray: 6d8578464e18b3ee-IAH
cf-ray: 6d8578464e18b3ee-IAH
< alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

< 
* Connection #0 to host bw.my-domain.com left intact

And it’s the same result for anything other than the subdomain for Nextcloud.

I have a PFsense box in front of it getting DDNS from Cloudflare at the root domain and the “www.” subdomain, and it’s pulling my IP just fine.

Same thing with the “unraid” subdomain to try and access my Unraid instance.

Any help will be greatly appreciated.