SSL causing website to not load at all

Okay so…

skylon.com and www.skylon.com have no AAAA records, and are the ones we have no issues with.

test.skylon.com has an AAAA record. How?! There are absolutely no AAAA records in my CF account, only A:

;; A Records (IPv4 addresses)
www.skylon.com. 1 IN A 23.253.218.146
skylon.com. 1 IN A 23.253.218.146
test.skylon.com. 1 IN A 23.253.218.146

The only other records are MX, TXT, and CNAME.

The AAAA records are automatically assigned (along the standard A records) when you proxy through Cloudflare. Proxied sites are reachable via IPv6 even when the origin does not support IPv6.

You’d need to disable IPv6 on Cloudflare’s side to remove these. Though that requires a call via the API as Cloudflare does not offer the ability to disable it via the UI.

1 Like

Okay, cool. That’s done, and I verified the response from the API:

{“result”:{“id”:“ipv6”,“value”:“off”,“modified_on”:“2018-10-17T16:45:56.586227Z”,“editable”:true},“success”:true,“errors”:,“messages”:}

Still seeing the issue, but I am thinking the user might need time for the TTL to die on that record, so I will leave it for a bit. Thank you for all your help, and I’ll update if it works or not.

Though the entire issue seems a bit weird. The original error message would leave the impression Cloudflare cant be reached at all, otherwise there should be at least a Cloudflare error. However the trace page does show the request goes through.

Unless iOS’ Safari (and yes, Chrome is just a Safari shell on iOS) does some error hiding (like IE used to do for some errors).

What about that?

Plain http works as intended everywhere. Site loading perfectly. My co-worker is on lunch, but I am going to have her test the https://test.* again once she’s back.

Hi @paul33, question and a suggestion.

Question - do the issues with iOS device happen only on the mobile network or on wifi as well? If only mobile, only Rogers or other operators, too?
Suggestion - As you trouble shoot this with your colleague, on the Network tab, turn off HTTP/2?

If /cdn-cgi/ hadnt worked, I would have pointed towards TLS. But with it working :confused:

I turned off HTTP2, and it is working on both wifi and Rogers now. Does this mean we cannot use http2, because iOS/Safari is having issues with it?

EDITED because I can no longer reply? I’ve reached my limit…

So…

try this set TLS minimum to TLS v1.2 and re-enable HTTP/2

Done, back to not being able to reload page.

Removed h2c from protocols on origin apache server. Do I need to refresh cache?

:flushed:

try this set TLS minimum to TLS v1.2 and re-enable HTTP/2

edit also if using Apache origin web server make sure h2c isn’t set in your protocol supported list

Protocols h2 h2c http/1.1

so maybe

Protocols h2 http/1.1

Can we please go back to unencrypted HTTP 1.0 (who cares about host headers)? Things were so much easier back then :wink:

</rant>

@cloonan, any idea why that broke on Safari, but only for the proxied page and not cdn-cgi?

This was bothering me so I dug into old support tickets and found a comment on a similar ticket to head of line blocking that pointed to this article, https://engineering.salesforce.com/the-full-picture-on-http-2-and-hol-blocking-7f964b34d205.

This line makes me think it’s latency over the mobile network, “…unexpectedly large variations in latency or a loss affecting the segment at the TCP head of line makes HTTP/2 more likely to hit HOLB, and its impact is larger as well. Basically, the receiver stands idle till the head is recovered, while all the subsequent segments are held by the TCP stack. This means that an image might be downloaded successfully and still not show up due to HOLB.”

1 Like

yeah i’d purge cf cache globally and verify Cloudflare proxied requests are not serving Upgrade header in your requests

also ensure you restarted apache after making the change to remove h2c

Purged, and yes, I’ve restarted apache a dozen times. :slight_smile:

When I check the headers of https://test.*, it’s showing me an upgrade header: “upgrade: http/1.”

During testing this afternoon, I found out that I can replicate the issue with cURL!

With http2 enabled:

[email protected]:/etc/apache2$ curl -v --http2 https://test.skylon.com

  • Rebuilt URL to: https://test.skylon.com/
  • Trying 104.26.5.63…
  • TCP_NODELAY set
  • Connected to test.skylon.com (104.26.5.63) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=skylon.com
  • start date: Oct 15 00:00:00 2018 GMT
  • expire date: Oct 15 12:00:00 2019 GMT
  • subjectAltName: host “test.skylon.com” matched cert’s “*.skylon.com”
  • issuer: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-2
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x5559d76b1810)

GET / HTTP/2
Host: test.skylon.com
User-Agent: curl/7.58.0
Accept: /

  • Connection state changed (MAX_CONCURRENT_STREAMS updated)!
  • http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [http/1.]
  • HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
  • Connection #0 to host test.skylon.com left intact
    curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

Website never outputs.

With http2 disabled:

[email protected]:/etc/apache2$ curl -v --http2 https://test.skylon.com

  • Rebuilt URL to: https://test.skylon.com/
    % Total % Received % Xferd Average Speed Time Time Time Current
    Dload Upload Total Spent Left Speed
    0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0* Trying 104.26.4.63…
  • TCP_NODELAY set
  • Connected to test.skylon.com (104.26.4.63) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
    } [5 bytes data]
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
    } [213 bytes data]
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
    { [106 bytes data]
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
    { [2169 bytes data]
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
    { [114 bytes data]
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
    { [4 bytes data]
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
    } [37 bytes data]
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
    } [1 bytes data]
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
    } [16 bytes data]
  • TLSv1.2 (IN), TLS handshake, Finished (20):
    { [16 bytes data]
  • SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
  • ALPN, server accepted to use http/1.1
  • Server certificate:
  • subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=skylon.com
  • start date: Oct 15 00:00:00 2018 GMT
  • expire date: Oct 15 12:00:00 2019 GMT
  • subjectAltName: host “test.skylon.com” matched cert’s “*.skylon.com”
  • issuer: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-2
  • SSL certificate verify ok.
    } [5 bytes data]

GET / HTTP/1.1
Host: test.skylon.com
User-Agent: curl/7.58.0
Accept: /

0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Thu, 18 Oct 2018 01:56:20 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=d829126e731937d1d656de95c466eb0411539827779; expires=Fri, 18-Oct-19 01:56:19 GMT; path=/; domain=.skylon.com; HttpOnly
< Link: https://www.skylon.com/wp-json/; rel=“https://api.w.org/”, https://www.skylon.com/; rel=shortlink
< Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
< Upgrade: http/1.
< Cache-Control: max-age=600
< Expires: Thu, 18 Oct 2018 02:06:19 GMT
< Vary: Accept-Encoding
< Expect-CT: max-age=604800, report-uri=“https://report-uri.Cloudflare.com/cdn-cgi/beacon/expect-ct
< Server: Cloudflare
< CF-RAY: 46b76345aa7955f4-ORD
<
{ [639 bytes data]
100 90670 0 90670 0 0 71113 0 --:–:-- 0:00:01 --:–:-- 71057

And I get an output of the html code as well (which I’ve omitted.)

Does this shed any light on the issue?

Thanks to all who have been helping me :slight_smile:

problem as i suspected is still the rogue HTTP header = Upgrade HTTP/1 being sent from your origin and being passed on by Cloudflare it seems as you can’t upgrade an already HTTP/2 connection to HTTP/2

http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [http/1.]
HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Connection #0 to host test.skylon.com left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

check all Apache origin httpd.conf and apache vhost and .htaccess files for any additionally added Upgrade headers that maybe added

1 Like

Unfortunately, there’s no instance of the word “upgrade” in my /etc/apache, and I have no other config files.

Also, a curl -v -http2 http://www.skylon.com never shows the upgrade header, which makes me think it can’t be passed from the origin server? Wouldn’t it show on the origin server as well? (Which is running http2.)

try a recursive case insensitive grep search in /etc/apache

grep -ir 'upgrade' /etc/apache

can also do search on your apache public web root directories to see if .htaccess files added such

I did do a recursive search… HOWEVER. I fixed it.

My protocols line was

Protocols h2 http/1.

I am 100% unsure why it was like that, but the trailing 1 was missing and that killed it. I re-enabled http2, and ran a cURL again:

  • SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=skylon.com
  • start date: Oct 15 00:00:00 2018 GMT
  • expire date: Oct 15 12:00:00 2019 GMT
  • subjectAltName: host “test.skylon.com” matched cert’s “*.skylon.com”
  • issuer: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-2
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
    } [5 bytes data]
  • Using Stream ID: 1 (easy handle 0x561d833a0810)
    } [5 bytes data]

GET / HTTP/2
Host: test.skylon.com
User-Agent: curl/7.58.0
Accept: /

{ [5 bytes data]

  • Connection state changed (MAX_CONCURRENT_STREAMS updated)!
    } [5 bytes data]
    0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0< HTTP/2 200
    < date: Thu, 18 Oct 2018 02:14:14 GMT
    < content-type: text/html; charset=UTF-8
    < set-cookie: __cfduid=df7efc7e273272d29e3efb9d89ed110211539828852; expires=Fri, 18-Oct-19 02:14:12 GMT; path=/; domain=.skylon.com; HttpOnly
    < link: https://www.skylon.com/wp-json/; rel=“https://api.w.org/”, https://www.skylon.com/; rel=shortlink
    < strict-transport-security: max-age=63072000; includeSubdomains; preload
    < cache-control: max-age=600
    < expires: Thu, 18 Oct 2018 02:24:12 GMT
    < vary: Accept-Encoding
    < expect-ct: max-age=604800, report-uri=“https://report-uri.Cloudflare.com/cdn-cgi/beacon/expect-ct
    < server: Cloudflare
    < cf-ray: 46b77d793f4d5594-ORD
    <
    { [942 bytes data]
    100 90670 0 90670 0 0 43094 0 --:–:-- 0:00:02 --:–:-- 43114
  • Connection #0 to host test.skylon.com left intact

I went back through and re-enabled stuff in CF I had disabled (like TLS 1.3), and I will get my co-workers to test tomorrow.

Thank you SO MUCH for the help!!!

1 Like

ah typo ! heh

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.