paul33
October 17, 2018, 4:34pm
#15
Okay so…
skylon.com and www.skylon.com have no AAAA records, and are the ones we have no issues with.
test.skylon.com has an AAAA record. How?! There are absolutely no AAAA records in my CF account, only A:
;; A Records (IPv4 addresses)
www.skylon.com . 1 IN A 23.253.218.146
skylon.com . 1 IN A 23.253.218.146
test.skylon.com . 1 IN A 23.253.218.146
The only other records are MX, TXT, and CNAME.
sandro
October 17, 2018, 4:38pm
#16
The AAAA records are automatically assigned (along the standard A records) when you proxy through Cloudflare. Proxied sites are reachable via IPv6 even when the origin does not support IPv6.
You’d need to disable IPv6 on Cloudflare’s side to remove these. Though that requires a call via the API as Cloudflare does not offer the ability to disable it via the UI.
1 Like
paul33
October 17, 2018, 4:48pm
#17
Okay, cool. That’s done, and I verified the response from the API:
{“result”:{“id”:“ipv6”,“value”:“off”,“modified_on”:“2018-10-17T16:45:56.586227Z”,“editable”:true},“success”:true,“errors”: ,“messages”: }
Still seeing the issue, but I am thinking the user might need time for the TTL to die on that record, so I will leave it for a bit. Thank you for all your help, and I’ll update if it works or not.
sandro
October 17, 2018, 4:52pm
#18
Though the entire issue seems a bit weird. The original error message would leave the impression Cloudflare cant be reached at all, otherwise there should be at least a Cloudflare error. However the trace page does show the request goes through.
Unless iOS’ Safari (and yes, Chrome is just a Safari shell on iOS) does some error hiding (like IE used to do for some errors).
sandro:
Does plain HTTP work?
What about that?
paul33
October 17, 2018, 5:30pm
#19
Plain http works as intended everywhere. Site loading perfectly. My co-worker is on lunch, but I am going to have her test the https://test .* again once she’s back.
cloonan
October 17, 2018, 5:36pm
#20
Hi @paul33 , question and a suggestion.
Question - do the issues with iOS device happen only on the mobile network or on wifi as well? If only mobile, only Rogers or other operators, too?
Suggestion - As you trouble shoot this with your colleague, on the Network tab, turn off HTTP/2?
sandro
October 17, 2018, 5:38pm
#21
If /cdn-cgi/
hadnt worked, I would have pointed towards TLS. But with it working
paul33
October 17, 2018, 5:53pm
#22
I turned off HTTP2, and it is working on both wifi and Rogers now. Does this mean we cannot use http2, because iOS/Safari is having issues with it?
EDITED because I can no longer reply? I’ve reached my limit…
So…
try this set TLS minimum to TLS v1.2 and re-enable HTTP/2
Done, back to not being able to reload page.
Removed h2c from protocols on origin apache server. Do I need to refresh cache?
eva2000
October 17, 2018, 6:08pm
#24
try this set TLS minimum to TLS v1.2 and re-enable HTTP/2
edit also if using Apache origin web server make sure h2c isn’t set in your protocol supported list
Protocols h2 h2c http/1.1
so maybe
Protocols h2 http/1.1
sandro
October 17, 2018, 6:17pm
#25
Can we please go back to unencrypted HTTP 1.0 (who cares about host headers)? Things were so much easier back then
</rant>
@cloonan , any idea why that broke on Safari, but only for the proxied page and not cdn-cgi
?
cloonan
October 17, 2018, 6:28pm
#26
sandro:
any idea
This was bothering me so I dug into old support tickets and found a comment on a similar ticket to head of line blocking that pointed to this article, The Full Picture on HTTP/2 and HOL Blocking | by Shauli Gal | Salesforce Engineering .
This line makes me think it’s latency over the mobile network, “…unexpectedly large variations in latency or a loss affecting the segment at the TCP head of line makes HTTP/2 more likely to hit HOLB, and its impact is larger as well. Basically, the receiver stands idle till the head is recovered, while all the subsequent segments are held by the TCP stack. This means that an image might be downloaded successfully and still not show up due to HOLB.”
1 Like
eva2000
October 17, 2018, 8:02pm
#27
yeah i’d purge cf cache globally and verify Cloudflare proxied requests are not serving Upgrade header
in your requests
also ensure you restarted apache after making the change to remove h2c
paul33
October 18, 2018, 1:57am
#28
Purged, and yes, I’ve restarted apache a dozen times.
When I check the headers of https://test .*, it’s showing me an upgrade header: “upgrade: http/1.”
During testing this afternoon, I found out that I can replicate the issue with cURL!
With http2 enabled:
[email protected] :/etc/apache2$ curl -v --http2 https://test.skylon.com
Rebuilt URL to: https://test.skylon.com/
Trying 104.26.5.63…
TCP_NODELAY set
Connected to test.skylon.com (104.26.5.63) port 443 (#0 )
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
TLSv1.2 (OUT), TLS handshake, Client hello (1):
TLSv1.2 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Client hello (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
ALPN, server accepted to use h2
Server certificate:
subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=skylon.com
start date: Oct 15 00:00:00 2018 GMT
expire date: Oct 15 12:00:00 2019 GMT
subjectAltName: host “test.skylon.com ” matched cert’s “*.skylon.com”
issuer: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-2
SSL certificate verify ok.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
Using Stream ID: 1 (easy handle 0x5559d76b1810)
GET / HTTP/2
Host: test.skylon.com
User-Agent: curl/7.58.0
Accept: /
Connection state changed (MAX_CONCURRENT_STREAMS updated)!
http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [http/1.]
HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Connection #0 to host test.skylon.com left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Website never outputs.
With http2 disabled:
[email protected] :/etc/apache2$ curl -v --http2 https://test.skylon.com
Rebuilt URL to: https://test.skylon.com/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0* Trying 104.26.4.63…
TCP_NODELAY set
Connected to test.skylon.com (104.26.4.63) port 443 (#0 )
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [213 bytes data]
TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2169 bytes data]
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [114 bytes data]
TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
TLSv1.2 (OUT), TLS change cipher, Client hello (1):
} [1 bytes data]
TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
ALPN, server accepted to use http/1.1
Server certificate:
subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=skylon.com
start date: Oct 15 00:00:00 2018 GMT
expire date: Oct 15 12:00:00 2019 GMT
subjectAltName: host “test.skylon.com ” matched cert’s “*.skylon.com”
issuer: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-2
SSL certificate verify ok.
} [5 bytes data]
GET / HTTP/1.1
Host: test.skylon.com
User-Agent: curl/7.58.0
Accept: /
0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0{ [5 bytes data]
< HTTP/1.1 200 OK
< Date: Thu, 18 Oct 2018 01:56:20 GMT
< Content-Type: text/html; charset=UTF-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Set-Cookie: __cfduid=d829126e731937d1d656de95c466eb0411539827779; expires=Fri, 18-Oct-19 01:56:19 GMT; path=/; domain=.skylon.com ; HttpOnly
< Link: https://www.skylon.com/wp-json/ ; rel=“https://api.w.org/ ”, https://www.skylon.com/ ; rel=shortlink
< Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
< Upgrade: http/1.
< Cache-Control: max-age=600
< Expires: Thu, 18 Oct 2018 02:06:19 GMT
< Vary: Accept-Encoding
< Expect-CT: max-age=604800, report-uri=“https://report-uri.Cloudflare.com/cdn-cgi/beacon/expect-ct ”
< Server: Cloudflare
< CF-RAY: 46b76345aa7955f4-ORD
<
{ [639 bytes data]
100 90670 0 90670 0 0 71113 0 --:–:-- 0:00:01 --:–:-- 71057
And I get an output of the html code as well (which I’ve omitted.)
Does this shed any light on the issue?
Thanks to all who have been helping me
eva2000
October 18, 2018, 1:59am
#29
paul33:
http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [http/1.]
HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Connection #0 to host test.skylon.com left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
problem as i suspected is still the rogue HTTP header = Upgrade HTTP/1 being sent from your origin and being passed on by Cloudflare it seems as you can’t upgrade an already HTTP/2 connection to HTTP/2
http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [http/1.]
HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
Connection #0 to host test.skylon.com left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
check all Apache origin httpd.conf and apache vhost and .htaccess files for any additionally added Upgrade headers that maybe added
1 Like
paul33
October 18, 2018, 2:07am
#30
Unfortunately, there’s no instance of the word “upgrade” in my /etc/apache, and I have no other config files.
Also, a curl -v -http2 http://www.skylon.com never shows the upgrade header, which makes me think it can’t be passed from the origin server? Wouldn’t it show on the origin server as well? (Which is running http2.)
eva2000
October 18, 2018, 2:13am
#31
paul33:
/etc/apache
try a recursive case insensitive grep search in /etc/apache
grep -ir 'upgrade' /etc/apache
can also do search on your apache public web root directories to see if .htaccess files added such
paul33
October 18, 2018, 2:23am
#32
I did do a recursive search… HOWEVER. I fixed it.
My protocols line was
Protocols h2 http/1.
I am 100% unsure why it was like that, but the trailing 1 was missing and that killed it. I re-enabled http2, and ran a cURL again:
SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
ALPN, server accepted to use h2
Server certificate:
subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=skylon.com
start date: Oct 15 00:00:00 2018 GMT
expire date: Oct 15 12:00:00 2019 GMT
subjectAltName: host “test.skylon.com ” matched cert’s “*.skylon.com”
issuer: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-2
SSL certificate verify ok.
Using HTTP2, server supports multi-use
Connection state changed (HTTP/2 confirmed)
Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
Using Stream ID: 1 (easy handle 0x561d833a0810)
} [5 bytes data]
GET / HTTP/2
Host: test.skylon.com
User-Agent: curl/7.58.0
Accept: /
{ [5 bytes data]
Connection state changed (MAX_CONCURRENT_STREAMS updated)!
} [5 bytes data]
0 0 0 0 0 0 0 0 --:–:-- 0:00:01 --:–:-- 0< HTTP/2 200
< date: Thu, 18 Oct 2018 02:14:14 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: __cfduid=df7efc7e273272d29e3efb9d89ed110211539828852; expires=Fri, 18-Oct-19 02:14:12 GMT; path=/; domain=.skylon.com ; HttpOnly
< link: https://www.skylon.com/wp-json/ ; rel=“https://api.w.org/ ”, https://www.skylon.com/ ; rel=shortlink
< strict-transport-security: max-age=63072000; includeSubdomains; preload
< cache-control: max-age=600
< expires: Thu, 18 Oct 2018 02:24:12 GMT
< vary: Accept-Encoding
< expect-ct: max-age=604800, report-uri=“https://report-uri.Cloudflare.com/cdn-cgi/beacon/expect-ct ”
< server: Cloudflare
< cf-ray: 46b77d793f4d5594-ORD
<
{ [942 bytes data]
100 90670 0 90670 0 0 43094 0 --:–:-- 0:00:02 --:–:-- 43114
Connection #0 to host test.skylon.com left intact
I went back through and re-enabled stuff in CF I had disabled (like TLS 1.3), and I will get my co-workers to test tomorrow.
Thank you SO MUCH for the help!!!
1 Like
system
Closed
November 17, 2018, 2:28am
#34
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.