WSS Closing - Only when proxied via Cloudflare

Our application site (https://sonorancad.com) has a small number of users that experience frequent web socket disconnections/reconnections. There doesn’t appear to be any consistency in the timing.

I know that CF will terminate/close websockets that are idle for ~2 minutes? We ensure that a keepalive is sent if there’s no active traffic.

Only a few clients experience this issue, but we can confirm it’s only with Cloudflare’s proxy. When we direct them to https://nocf.sonorancad.com (same site - no CF proxying) they don’t have any connection issues.

Is there anything on the CF configuration end that might cause some websockets to be randomly terminated? The user gets a standard 1006 error code.

Update:

I enabled some trace logging and was able to get more logs from a client experiencing this issue. The client sends a SignalR type 6 (KeepAlive) message to the server, and immediately has the WSS connection closed.

I’m not sure what would cause the socket to close in exact response to that keepalive being sent, but it only occurs when the connection is proxied through CF.

Could this message type be seen as some sort of spam, being filtered by CF?
image
image

@MoreHelp #2249343

Cloudflare regularly releases new code to its server fleetS, causing server restarts, which is a common cause of WS disconnections. Cloudflare recommends having built-in reconnection logic for this.

When Cloudflare releases new code to its global network, we may restart servers, which terminates WebSockets connections.

This would certainly explain a few network blips here and there, and we do have built in reconnection logic.

However, the primary issue is with certain users constantly being disconnected and reconnected, making the service almost entirely unusable until they use a non-cf proxied version of our app.

With it being disconnected almost every few seconds, my guess is some sort of firewall or rate limiting almost on CF’s end possibly detecting their reconnection logic as malicious?

That’s indeed possible, but in that case you would see new events related to that in your firewall log, which can be seen at the Cloudflare dashboard.

If you have not already, I would perhaps add a heartbeat, so you basically send at least a WS message every 10 seconds or so, as it is possible that Cloudflare terminates idle websockets after a while.

Yes, there is a keepalive heartbeat. The screenshot logs also show traffic being sent constantly back and forth, so these websockets are not idle.

The WS is specifically closed when the heartbeat is sent from client → server, as shown in the screenshot. These keepalive heartbeats are sent every 15 seconds.

Someone else had also suggested that the WS was already closed by CF, but the client doesn’t realize this until the keepalive message is sent - thus the “instant” closure after trying to send the next message.

The question is in why CF is closing these websockets, they’re by no means idle.

Try sending heartbeat messages every second (or maybe 2-3s), it still disconnects?

Never experienced this kind of issue.

This would be a fairly incredible amount of web traffic for several thousand concurrent clients.

I could increase the frequency of the keep alive/heartbeat pings to every 10 seconds, but there’s no reason why CF would be closing “idle” websocket connections after just 15 seconds.

The main issue is these users being constantly disconnected every few seconds. It’s not just a rare disconnection every so often.

have you check your server logs? anything from there?