Cloudflare SSL issues for end user, again

cvagnetoft · February 5, 2021, 4:01pm

Turns out I was too quick to celebrate things working in the last post I created, the issues are back:

*   Trying 104.26.15.6:443...
* TCP_NODELAY set
* Connected to flex.symfony.com (104.26.15.6) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=symfony.com
*  start date: Jul  2 00:00:00 2020 GMT
*  expire date: Jul  2 12:00:00 2021 GMT
*  subjectAltName: host "flex.symfony.com" matched cert's "*.symfony.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564a896cee80)
> GET /versions.json HTTP/2
> Host: flex.symfony.com
> user-agent: curl/7.68.0
> accept: */*
> 
* OpenSSL SSL_read: Connection reset by peer, errno 104
* Failed receiving HTTP2 data
* OpenSSL SSL_write: Broken pipe, errno 32
* Failed sending HTTP2 data
* Connection #0 to host flex.symfony.com left intact
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104

Affected sites now include *.symfony.com, vaping360.com, cdnjs.cloudflare.com (but not community or main site)

The other thread (Can't reach cdnjs, most other Cloudflare sites work intermittently) was closed, so had to open a new one. The issue is basically the same tho.

sdayman · February 5, 2021, 4:21pm

What’s the command you’re running? I can’t help but notice it’s trying TLS 1.3, which doesn’t always work with some clients.

cvagnetoft · February 5, 2021, 6:56pm

I’ve tested with curl, php fopen, Google Chrome, and a few other browsers. It also affects other devices on the network. That particular output is from curl -vvv https://flex.symfony.com

Deimos_Col · February 5, 2021, 7:44pm

Hello, the following devices will have connection problems, if you define TLS 1.3:

By enabling TLS 1.0:

More devices will be able to connect, as it is less secure.

cvagnetoft · February 5, 2021, 8:33pm

It is still using TLS1.3 when it works:

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare.com
*  start date: Jul  4 00:00:00 2020 GMT
*  expire date: Jul  4 12:00:00 2021 GMT
*  subjectAltName: host "community.cloudflare.com" matched cert's "*.cloudflare.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55d85a284e80)
} [5 bytes data]
> GET / HTTP/2
> Host: community.cloudflare.com
> user-agent: curl/7.68.0
> accept: */*
> 
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 200 
< date: Fri, 05 Feb 2021 20:30:54 GMT
< content-type: text/html; charset=utf-8
...

The error is intermittent. 2 of 3 curl requests succeeded just now, with this one in between. All are for the same domain community.cloudflare.com.

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare.com
*  start date: Jul  4 00:00:00 2020 GMT
*  expire date: Jul  4 12:00:00 2021 GMT
*  subjectAltName: host "community.cloudflare.com" matched cert's "*.cloudflare.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55f9ed05fe80)
} [5 bytes data]
> GET / HTTP/2
> Host: community.cloudflare.com
> user-agent: curl/7.68.0
> accept: */*
> 
* OpenSSL SSL_read: Connection reset by peer, errno 104

fritex · February 5, 2021, 10:42pm

Is this related to your browser settings or?

Firefox has “TRR mode”. If you set it up too high (1-4) like 4, you would not open any Website which is not using TLS v1.3.
Similar, value 3, is the TLS v.1.2.

If the SSL certificate ciphers are not in the line with TLS version, and your Web browser is somehow not supporting some of it, I suppose the errors would show up.

Keep up with TLS v1.2 and have an option enabled for TLS v1.3 (if the user’s web browser supports it and accepts it).

Recently, we moved from 1.0 and 1.1 to 1.2.
But, there are also a lot of clients and services still running and have SSL certificate generated with supported ciphers for 1.0 and 1.1.

In the meanwhile in some cases I saw even goverment websites and apps working only on 1.0 and 1.1. So they lack the 1.2 and not even to mention about 1.3 support if your/users Web browser supports the minimum v1.2 and no lower version than that.

cvagnetoft · February 5, 2021, 10:52pm

I doubt it has to do with browser settings either. cdnjs.cloudflare.com was unavailable for quite a while, and then it worked for a couple of days. Now its intermittent again, together with other affected sites, although the situation isn’t quite as bad as the last round. Also, android smartphone and tablet as well as smart tv has problems with the same sites.

fritex · February 5, 2021, 11:00pm

Well, I have one Xiaomi Redmi 1S which is Android 4.4.4. That and any lower Android OS version would have an issue opening a Website which works only on TLS v.1.3 for sure.

cvagnetoft · February 5, 2021, 11:06pm

The websites have worked in the past, and work intermittently now. The only change in my end is a new IP address.

fritex · February 5, 2021, 11:08pm

Hm, strange.
Saying, you changed the ISP or your IP at home changed due to for example. DHCP or ruter restart?
Are you using Cloudflare WARP or 1.1.1.1 in the DNS?
Have you tried clearing, if so, your DNS cache on devices or Web browser if you have accessed the Websites before?

cvagnetoft · February 5, 2021, 11:14pm

Same ISP, only went from DSL to 4G. Router, WiFi AP and everything else on the network has been restarted multiple times. The WiFi AP enforces 8.8.8.8, 8.8.4.4 while the router uses default - doesn’t make a difference which one you are connected to. Also, if the server is available over http (such as cdnjs) it responds over http but not https. So it feels like the routing/dns outside of CF is sane from where i am?

cvagnetoft · February 7, 2021, 1:25pm

So, being a bored dev I hacked this together. Here is 60 minutes of my Internet. One HEAD request per minute, . indicates the site was unreachable and ^ that it was up. The percentages at the end are interesting, as some sites are completely dead, some sites work intermittently, and some sites are down intermittently. This is so random it makes my head spin…

https://cdnjs.cloudflare.com                 ............................................................ Down since 23:44:31, 0 of 155 good (100.0% bad)
https://production.cloudflare.docker.com     ....^^^^^^^^.^^^^.^^^^...^.....^^^^....^....^..^^^^^^^^^.^^^ Up since 13:56:57, 74 of 135 good (45.2% bad)
https://mysteriousuniverse.org               ............................................................ Down since 23:44:32, 0 of 155 good (100.0% bad)
https://prisjakt.nu                          ^..^^....^......^^.^^^^.^.....^^.^..^....^^^^^^^^..^.....^^^ Up since 13:56:58, 65 of 149 good (56.4% bad)
https://symfony.com                          ............................................................ Down since 23:44:32, 0 of 154 good (100.0% bad)
https://vaping360.com                        ............................................^............... Down since 13:44:37, 2 of 154 good (98.7% bad)

So I can still not create any new Symfony projects (the symfony tool uses composer and adds their flex repository). I can kind of deploy new docker stacks, by being persistent and running the command till it succeds (unless it needs something else CF). This is affecting me working from home, and not a single official word from CloudFlare here.

Is it better to tweet all the affected sites and have them contact Cloudflare through the “Red Phone”? Cloudflare has been showing “All Systems Operational” for 2 weeks, while it is pretty obvious that something is wrong. Any ideas on how to make CF aware that something is wrong?

Connections are dropped after the SSL handshake has completed. I doubt my ISP or anything else along the way would attempt identify SSL frames just to drop connections at random, and since it affects the entire home network and 2 different wifi ap:s (both through the same connection tho.)

Any suggestions or ideas are welcome. If there is a workaround that lets me do my job without jumping hoops or delaying, or if anyone knows a way to get a hold of someone at CloudFlare that can actually investigate this from the inside that would be amazing.

cvagnetoft · February 8, 2021, 12:57am

This looks extremely promising! Not celebrating yet though

https://cdnjs.cloudflare.com                 .....................................................^^^^^^^ Up since 01:48:26 Mon 08 Feb, 44 of 734 good (94.0% bad)
https://production.cloudflare.docker.com     ..^..^^^^^.^^^^^^.^^^^.^^^^^^^^^^^.....^^^.^^^^......^^^^^^^ Up since 01:48:26 Mon 08 Feb, 454 of 714 good (36.4% bad)
https://mysteriousuniverse.org               .....................................................^^^^^^^ Up since 01:48:27 Mon 08 Feb, 24 of 734 good (96.7% bad)
https://prisjakt.nu                          ^^.^^^^^.^...^....^^^^^.^^^^^....^^^^^.^.....^^^...^^^^^^^^^ Up since 01:46:23 Mon 08 Feb, 351 of 728 good (51.8% bad)
https://symfony.com                          .....................................................^^^^^^^ Up since 01:48:27 Mon 08 Feb, 26 of 733 good (96.5% bad)
https://vaping360.com                        .....................................................^^^^^^^ Up since 01:48:28 Mon 08 Feb, 44 of 733 good (94.0% bad)

cvagnetoft · February 8, 2021, 12:02pm

And it is back to broken again, just in time for the morning coffee!

https://cdnjs.cloudflare.com                 ........................^^^^^^^^^^^^........................ Down since 02:01:17 Mon 08 Feb, 49 of 763 good (93.6% bad)
https://production.cloudflare.docker.com     ^^^^^.....^^^.^^^^......^^^^^^^^^^^^^^..^..^^^^^..^^^^^^.^^^ Up since 12:59:48 Mon 08 Feb, 476 of 743 good (35.9% bad)
https://mysteriousuniverse.org               ........................^^^^^^^^^^^^........................ Down since 02:01:18 Mon 08 Feb, 29 of 763 good (96.2% bad)
https://prisjakt.nu                          ....^^^^^.^.....^^^...^^^^^^^^^^^^^^..^^^...^^^^^^^^^.^....^ Up since 13:01:54 Mon 08 Feb, 370 of 757 good (51.1% bad)
https://symfony.com                          ........................^^^^^^^^^^^^........................ Down since 02:01:19 Mon 08 Feb, 31 of 762 good (95.9% bad)
https://vaping360.com                        ........................^^^^^^^^^^^^........................ Down since 02:01:20 Mon 08 Feb, 49 of 762 good (93.6% bad)

cvagnetoft · February 10, 2021, 12:10am

The problem persists, cdnjs and symfony has been unavailable all day.

For the sake of troubleshooting I’ve also tried (with curl) TLS1.1, TLS1.2 and TLS1.3 in combination with HTTP1.1 and HTTP2 and everything result in the same errno 104.

cvagnetoft · February 22, 2021, 1:05pm

Still problematic. Cdnjs mostly works, Community.cloudflare.com doesn’t. Symfony is down, Dockerhub is up.

Screenshot from 2021-02-22 13-50-41

cvagnetoft · February 22, 2021, 1:15pm

Curl and OpenSSL output for community.cloudflare.com – Chrome seems to show ERR_FAILED since the last update, but the error under the hood is the same.

noccy@noccy-laptop:~$ curl -vv https://community.cloudflare.com
*   Trying 104.16.133.229:443...
* TCP_NODELAY set
* Connected to community.cloudflare.com (104.16.133.229) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=cloudflare.com
*  start date: Jul  4 00:00:00 2020 GMT
*  expire date: Jul  4 12:00:00 2021 GMT
*  subjectAltName: host "community.cloudflare.com" matched cert's "*.cloudflare.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55ef6940df20)
> GET / HTTP/2
> Host: community.cloudflare.com
> user-agent: curl/7.68.0
> accept: */*
> 
* OpenSSL SSL_read: Connection reset by peer, errno 104
* Failed receiving HTTP2 data
* OpenSSL SSL_write: Broken pipe, errno 32
* Failed sending HTTP2 data
* Connection #0 to host community.cloudflare.com left intact
curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
noccy@noccy-laptop:~$ openssl s_client community.cloudflare.com:443
CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = CA, L = San Francisco, O = "Cloudflare, Inc.", CN = cloudflare.com
verify return:1
---
Certificate chain
 0 s:C = US, ST = CA, L = San Francisco, O = "Cloudflare, Inc.", CN = cloudflare.com
   i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
 1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
   i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE7zCCBJSgAwIBAgIQD9WF1Et9s39cPmU79LIm9zAKBggqhkjOPQQDAjBKMQsw
CQYDVQQGEwJVUzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEgMB4GA1UEAxMX
Q2xvdWRmbGFyZSBJbmMgRUNDIENBLTMwHhcNMjAwNzA0MDAwMDAwWhcNMjEwNzA0
MTIwMDAwWjBmMQswCQYDVQQGEwJVUzELMAkGA1UECBMCQ0ExFjAUBgNVBAcTDVNh
biBGcmFuY2lzY28xGTAXBgNVBAoTEENsb3VkZmxhcmUsIEluYy4xFzAVBgNVBAMT
DmNsb3VkZmxhcmUuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEG8yJrKWY
lPQG6J3KgKis4XeOY3/cL5qEQ+mP3zgl4ylipgX7/XJLiQ5361b8fJhj/UjIudFJ
0xL9ibOrs6JTDqOCAz4wggM6MB8GA1UdIwQYMBaAFKXON+rrsHUOlGeItEX62SQQ
h5YfMB0GA1UdDgQWBBTUxXwi2+exV6YsgMeQajzIlUvqnTBxBgNVHREEajBoghAq
LmNsb3VkZmxhcmUuY29tgg5jbG91ZGZsYXJlLmNvbYIUKi5kbnMuY2xvdWRmbGFy
ZS5jb22CFCouYW1wLmNsb3VkZmxhcmUuY29tghgqLnN0YWdpbmcuY2xvdWRmbGFy
ZS5jb20wDgYDVR0PAQH/BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEF
BQcDAjB7BgNVHR8EdDByMDegNaAzhjFodHRwOi8vY3JsMy5kaWdpY2VydC5jb20v
Q2xvdWRmbGFyZUluY0VDQ0NBLTMuY3JsMDegNaAzhjFodHRwOi8vY3JsNC5kaWdp
Y2VydC5jb20vQ2xvdWRmbGFyZUluY0VDQ0NBLTMuY3JsMEwGA1UdIARFMEMwNwYJ
YIZIAYb9bAEBMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8vd3d3LmRpZ2ljZXJ0LmNv
bS9DUFMwCAYGZ4EMAQICMHYGCCsGAQUFBwEBBGowaDAkBggrBgEFBQcwAYYYaHR0
cDovL29jc3AuZGlnaWNlcnQuY29tMEAGCCsGAQUFBzAChjRodHRwOi8vY2FjZXJ0
cy5kaWdpY2VydC5jb20vQ2xvdWRmbGFyZUluY0VDQ0NBLTMuY3J0MAwGA1UdEwEB
/wQCMAAwggEDBgorBgEEAdZ5AgQCBIH0BIHxAO8AdQD2XJQv0XcwIhRUGAgwlFaO
400TGTO/3wwvIAvMTvFk4wAAAXMbCD9JAAAEAwBGMEQCIGvGTxZeBYgdbs+vnp0c
9y45gwRMOZnKz4eE1d4LdagwAiARwrwO1jORk0C5/VTj+RPxK5HRwK169bwkdVL8
0pB7lAB2AFzcQ5L+5qtFRLFemtRW5hA3+9X6R9yhc5SyXub2xw7KAAABcxsIP38A
AAQDAEcwRQIhAMWV1BR5WmEVEfpfyBAJErAa7zbUmn9yqwLzdVDesRcuAiAePxqr
LJBQ5ysyisJOIt6GzfUOg1rDmsgmmdj7zz9lgDAKBggqhkjOPQQDAgNJADBGAiEA
42x0wUe7wq/QbXOvyaYX8IXvORQ0L+/cOM4TrVCEG9ECIQDaDyZCKGGpi+MqdDmy
dLIonPk4USKu95+gu0CQPQTglQ==
-----END CERTIFICATE-----
subject=C = US, ST = CA, L = San Francisco, O = "Cloudflare, Inc.", CN = cloudflare.com

issuer=C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2558 bytes and written 406 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
read:errno=104
noccy@noccy-laptop:~$

michael · February 22, 2021, 10:23pm

Reading through this thread, I would be fairly certain the issue is local to you.

What type of connection are you on?
Does this happen if you change the internet connection? (Such as changing from a home wired connection to a hot-spot off your phone, or from wireless to wired)
What OS is on your laptop?
Does your ISP provide native IPv4, or some kind of CGNAT, DS-Lite etc?
If you lower your MTU, to say 1200, does the issue go away?

cvagnetoft · February 23, 2021, 11:09am

I’m on 4G now. Same ISP, used to be on DSL but upgraded to 4G, using equipment provided by ISP. All devices on the network experience the issue. The same software works great when tunneled over socks, or via hotspot. Phone 4G works, Phone WiFi (ISP 4G) doesn’t. Some days are good, with everything working great, and then it goes dark again. Quite a bit of randomness involved.

Google and Twitter are completely unaffected, have been and are working 100%.

The configuration is stock meaning ISP would be to blame for mismatched MTU, but it doesn’t explain why it sometimes work, or why the connection is always dropped just after the SSL handshake is established. Done tech for over 20 years, and this might be the first problem I’ve encountered that might actually make me bald. Nothing should react to the SSL handshake completing OK by killing it, so my suspicion is that something should react and pass it on, but that something is not feeling entirely well.

system · March 7, 2021, 4:35pm

This topic was automatically closed after 30 days. New replies are no longer allowed.

Topic		Replies	Views
Intermittent TLS errors when accessing websites behind Cloudflare via IPv6 Website, Application, Performance	7	1368	January 10, 2025
SSL is not working , I am using free plan of Cloudflare Security	9	3264	February 27, 2021
Intermittent curl SSL issue to cloudflare domains Security	1	1742	April 11, 2023
SSL 525 error continues Security	10	702	August 28, 2023
Cloudflare not issue certificate Security	3	1400	October 26, 2022

Cloudflare SSL issues for end user, again

Related topics