Hi
I’m having a persistent issue with Cloudflare and websockets. We have a frontend application, running on the staging-app subdomain that is attempting to open a websocket to our API application at the staging-api subdomain. These are both proxied by CF. We have SSL Full (Strict) enabled with authenticated origin pulls, and our origins all use hostname-level Cloudflare Origin SSL certificates. The origin is technically a Digital Ocean load balancer, which is backed by an nginx-ingress within our K8s cluster. We’re running Ruby on Rails for our API, and Rails’ ActionCable is our websockets framework. HTTPS calls all work perfectly, HTTP requests function properly.
The browser websocket request looks like this:
GET /websocket?token=<redacted for privacy> HTTP/1.1
Host: staging-api.<redacted>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: https://staging-app.<redacted>
Sec-WebSocket-Protocol: actioncable-v1-json, actioncable-unsupported
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: <redacted>
DNT: 1
Connection: keep-alive, Upgrade
Cookie: _<redacted>_session=<redacted>
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-site
Sec-GPC: 1
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket
And this is the response in the browser from CF:
HTTP/1.1 525
Date: Wed, 29 Jun 2022 21:42:42 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: cf_use_ob=0; path=/; expires=Wed, 29-Jun-22 21:43:12 GMT
X-Frame-Options: SAMEORIGIN
Referrer-Policy: same-origin
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
CF-RAY: 7231d565caa97ab4-LAX
Server: cloudflare
Based on this curl dump, everything about the SSL certificate seems fine to me but I can’t say for certain:
curl -svoik /dev/null https://staging-api.<redacted> --connect-to ::<redacted> --cacert ~/Downloads/origin_ca_ecc_root\(3\).pem 2>&1
* Connecting to hostname: <redacted>
* Trying <redacted>:443...
* Connected to <redacted> (<redacted>) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* CAfile: /Users/adam/Downloads/origin_ca_ecc_root(3).pem
* CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: O=CloudFlare, Inc.; OU=CloudFlare Origin CA; CN=CloudFlare Origin Certificate
* start date: Jun 22 03:08:00 2022 GMT
* expire date: Jun 18 03:08:00 2037 GMT
* subjectAltName: host "staging-api.<redacted>" matched cert's "staging-api.<redacted>"
* issuer: C=US; ST=California; L=San Francisco; O=CloudFlare, Inc.; OU=CloudFlare Origin SSL ECC Certificate Authority
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: staging-api.<redacted>]
* h2h3 [user-agent: curl/7.84.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15c013800)
> GET / HTTP/2
> Host: staging-api.<redacted>
> user-agent: curl/7.84.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Wed, 29 Jun 2022 21:49:35 GMT
< content-type: text/html
< vary: Accept-Encoding
< x-frame-options: SAMEORIGIN
< x-xss-protection: 0
< x-content-type-options: nosniff
< x-download-options: noopen
< x-permitted-cross-domain-policies: none
< referrer-policy: strict-origin-when-cross-origin
< cache-control: no-cache
< x-request-id: 7989a0b4716d651982bcb13b5ddbce00
< x-runtime: 0.002655
< vary: Origin
< strict-transport-security: max-age=15724800; includeSubDomains
<
* Connection #0 to host <redacted> left intact
Here’s the openssl dump:
openssl s_client -servername staging-api.<redacted> -tlsextdebug -connect <redacted> -CAfile ~/Downloads/origin_ca_ecc_root\(3\).pem </dev/null | openssl x509 -noout -text
depth=1 C = US, ST = California, L = San Francisco, O = "CloudFlare, Inc.", OU = CloudFlare Origin SSL ECC Certificate Authority
verify return:1
depth=0 O = "CloudFlare, Inc.", OU = CloudFlare Origin CA, CN = CloudFlare Origin Certificate
verify return:1
DONE
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
19:a4:64:e5:ab:0b:3d:6a:9b:a4:1c:91:ad:f4:a0:3a:a0:54:a2:8c
Signature Algorithm: ecdsa-with-SHA256
Issuer: C=US, ST=California, L=San Francisco, O=CloudFlare, Inc., OU=CloudFlare Origin SSL ECC Certificate Authority
Validity
Not Before: Jun 22 03:08:00 2022 GMT
Not After : Jun 18 03:08:00 2037 GMT
Subject: O=CloudFlare, Inc., OU=CloudFlare Origin CA, CN=CloudFlare Origin Certificate
Subject Public Key Info:
Public Key Algorithm: id-ecPublicKey
Public-Key: (521 bit)
pub:
<redacted>
ASN1 OID: secp521r1
NIST CURVE: P-521
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Client Authentication, TLS Web Server Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
<redacted>
X509v3 Authority Key Identifier:
k<redacted>
Authority Information Access:
OCSP - URI:http://ocsp.cloudflare.com/origin_ecc_ca
X509v3 Subject Alternative Name:
DNS:staging-api.<redacted>
X509v3 CRL Distribution Points:
Full Name:
URI:http://crl.cloudflare.com/origin_ecc_ca.crl
Signature Algorithm: ecdsa-with-SHA256
<redacted>
I don’t see any SSL issues here, but maybe I’m missing something.
Our DO load balancer has the proxy protocol enabled, and the only forwarding rules are TCP forwarding rules to the ingress. The ingress terminates TLS and is what I assume CF is having SSL issues with.
In the nginx-ingress K8s ingress, there are no annotations except for setting the ingress class. I have these global nginx-ingress configs set:
{
"allow-snippet-annotations": "true",
"enable-brotli": "true",
"proxy-body-size": "100m",
"proxy-read-timeout": "3600",
"proxy-send-timeout": "3600",
"ssl-ecdh-curve": "auto",
"ssl-protocols": "TLSv1.3",
"use-gzip": "true",
"use-http2": "true",
"use-proxy-protocol": "true"
}
Can someone help me out please? I’m totally stuck. Thank you!