Cloudflare returns 502 on Firefox and Chrome but works on Safari for web socket?

[northern.snow.x]

My Nginx runs as a reverse proxy and connects to Argo Tunnel.
The configuration looks like this:

Nginx:

server {
    server_name xxx;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/xxx/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/xxx/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
    location / {
        proxy_set_header Host $host;
	proxy_redirect off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection Upgrade;
        proxy_pass https://xx.xx.xx.xx;
    }
}
server {
    if ($host = xxx) {
        return 301 https://$host$request_uri;
    }

    listen 80;
    server_name xxx;
    return 404;
}

Cloudflared

ingress:

        - hostname: xxx
          service: https://localhost
          originRequest:
                  noTLSVerify: true
        - service: http_status:404

When browsers try to request web socket through wss://xxx/wss, Safari gets a success response of 101 Switching Protocols while Chrome and Firefox both get 502 Bad Gateway.

[chungting]

Hi @northern.snow.x,

Did the tunnel return 502 Bad Gateway to Chrome and Firefox? If you set log level to debug, you can see the response headers returned through the tunnel. Perhaps the tunnel didn’t return certain headers to complete the websocket handshake.

[northern.snow.x]

Hi,

The tunnel returns 502 Bad Gateway to Chrome and Firefox but not Safari.
I noticed subtle difference in Sec-Websocket-Extensions:

Working:

Request Headers map[Accept-Encoding:[gzip] Cache-Control:[no-cache] Cdn-Loop:[cloudflare] 
Sec-Websocket-Extensions:[x-webkit-deflate-frame]
User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.2 Safari/605.1.15]

Non-working:

Request Headers map[Accept:[*/*] Accept-Encoding:[gzip] Accept-Language:[en-US,zh-HK;q=0.7,zh-CN;q=0.3] Cache-Control:[no-cache] 
Sec-Websocket-Extensions:[permessage-deflate]
User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 11.1; rv:86.0) Gecko/20100101 Firefox/86.0] 

I don’t notice any other difference.

I dsiabled permessage-deflate and disabled Accept-Language. It still doesn’t work.

I found that overlooked this option: Early-Data:[1] It seems to be set by Cloudflare. Requests from Firefox come with this one. Safari doesn’t.

[northern.snow.x]

I think the Early-Data flag might be the root cause of my problem. Is it possible to turn it off from Cloudflare?

• I turned the flag off by disabling 0-RTT. However it still doesn’t help.

This is so weird and I have no idea what happened. If I don’t use Argo Tunnel but use direct conenction then all browsers work. If I use Argo Tunnel then only Safari works.

[northern.snow.x]

I turned on nginx’s debug mode. Nginx returns 400 Bad Request for incoming connection from Firefox with Cloudflare.

OK this is weird. Somehow nginx only receives the __cfduid cookie. Other cookies disappear! This is likely a bug.

[northern.snow.x]

Definitely a bug! It seems Argo Tunnel stops processing cookies once it meets CF_Authorization. The cookie sequence depends on the browser. That explains why Safari works but Firefox doesn’t. In Safari, Cloudflare cookies come after the necessary cookies for my site to function. However, in Firefox, Cloudflare cookies come first. When Argo drops rest of my cookies, the request is bad.

Log:

Good one:

[debug] 34353#34353: *7518 http proxy header: "Cookie: UserPref=kbd_type%3Den; irc=last%3Dhtml5; sessionLang=en; sessionKey=XXX; sessionUrl=XXX; CF_Authorization=XXX; __cfduid=XXX"
...
[debug] 34353#34353: *7518 http proxy header:
...
Cf-Warp-Tag-Id: XXX
Cookie: UserPref=kbd_type%3Den; irc=last%3Dhtml5; sessionLang=en; sessionKey=XXX; sessionUrl=XXX; CF_Authorization=XXX
[debug] 34353#34353: *7518 http cleanup add: 0000563A13171070

Bad one:

[debug] 34353#34353: *7248 http proxy header: "Cookie: __cfduid=XXX; CF_Authorization=XXX; sessionUrl=XXX; sessionLang=en; sessionKey=XXX; irc=last%3Dhtml5; UserPref=kbd_type%3Den"
...
[debug] 34353#34353: *7248 http proxy header:
...
Cf-Warp-Tag-Id: XXX
Cookie: __cfduid=XXX; CF_Authorization=XXX
[debug] 34353#34353: *7248 http cleanup add: 0000563A130D3930

[chungting]

Argo Tunnel doesn’t support websocket per message deflate at the moment, and it also doesn’t inspect cookies, so they should all be forward to your origin. Does Nginx expect cookies in a certain order?

[northern.snow.x]

Hi I’ve finally resolved the issue. It’s not a bug of cloudflared. The problem is that I passed the huge JWT token and Authorization cookies to the upstream, which is unnecessary and in this case, causes a corrupted header. After striping away all the headers set by cloudflare, the problem resolved.

[chungting]

I’m glad it’s working for you!