TLS handshake suddenly fails

Zero Trust Cloudflare Tunnel
1

I have a tunnel that ran smoothly but failed suddenly, and I’m trying to figure out why.

For some reason I can’t post url - using dummy strings.

The tunnel has encryption set to flexible, listens to 80 port of nginx at local and exposes the service to prefixDOTmydomainDOTcom . I rely on the tunnel to secure data in transit and has no TLS configure of my own. The service runs on the same box. The nginx config is the following:

server {
    listen 80;
    server_name prefixDOTmydomainDOTcom;
    location / {
        proxy_pass localhost9995;
        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        add_header Cache-Control no-cache;
    }

}

This worked fine for at least two weeks, during which I have done system upgrades and reboots, after which everything comes back without problem.

Today, when I try to connect, I got PR_END_OF_FILE_ERROR. Interestingly, all parts in this chain seem to report themselves as healthy:

  • dig and ping both return things
  • tunnel is listed as healthy in webUI
  • tunnel log saw no traffic at all when I start to log them in webUI
  • systemctl status nginx and /var/log/nginx/error.log has nothing
  • lastly, the service itself runs fine, I can access it locally or from the same LAN as expected.

The cloudflared is version version 2023.7.3 (built 20230811-02:13:50). The nginx also serves a webDAV at 80 for server name prefix2DOTmydomainDOTcom, which also worked and now has stopped working.

I have restarted cloudflared/nginx/myservice, but saw no errors. I am about 80% sure that I did nothing between the last successful connection to prefixDOTmydomainDOTcom and the current failure. (The other 20% is I did full system upgrade but haven’t reboot yet.)

I’m not sure how to investigate TLS handshake, but here is the output from openssl s_client -showcerts -connect prefixDOTmydomainDOTcom443 -prexit:

CONNECTED(00000003)
403C5B560D7F0000:error:0A000126:SSL routines:ssl3_read_n:unexpected eof while reading:ssl/record/rec_layer_s3.c:303:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 354 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 354 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Thank you for your time.

4

update: I rebooted the machine and problem persists.

5

Update: recreated the tunnel and updated cloudflared to current (2023.8.2-1), did full system update & fresh reboot, but problem persists. MWE:

server {
        listen 80;
        server_name cftestDOTdomainDOTdev;

        location / {
        default_type text/html;
        return 200 "Hello, world!";
    }
}

The above fails at TLS handshake. I still have no clue.