Facing RST_STREAM with error code: INTERNAL_ERROR on gRPC Server Side Streaming

Hello Cloudflare Support,

We’re currently encountering an intermittent issue with our gRPC communications, which are routed through Cloudflare. Our setup consists of two Golang applications that communicate via gRPC, and we’ve ensured that the gRPC check is enabled for our domain on Cloudflare.

The communication works seamlessly for the most part. However, we’ve observed that during some gRPC server-side streaming operations, there could be a prolonged period of inactivity (lasting a few minutes) with no messages being exchanged. Eventually, this inactivity leads to the termination of the stream, and we’re presented with the following error message:

error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

This issue does not happen consistently but seems to occur sporadically, especially noticeable during those longer pauses in the stream’s activity.

We have checked both our server and client application logs for any anomalies that might explain this behavior but have found none. Our server configurations have been reviewed to ensure they are optimized for long-duration gRPC connections.

We’re reaching out to seek your insights on what might be causing these RST_STREAM terminations. Are there any specific settings or conditions within Cloudflare that we should be aware of, which might trigger such an error, particularly during extended periods of inactivity in a gRPC stream? Any guidance or suggestions on configurations that might help avoid this issue would be greatly appreciated.

Thank you in advance for your assistance.

Hi @rqew2133gdfsa

gRPC only work on proxied endpoints, and you do not have any proxied DNS records, so this functionality will not work as it is right now.

Cloudflare typically has a 60-second idle timeout, after which it will terminate a connection if no data is sent. To address this, you can:

  1. Implement application-level keepalive messages to ensure the connection is not considered idle.
  2. Adjust your server’s keepalive settings to send pings more frequently.

Also the domain in your account is returning DNS_PROBE_FINISHED_NXDOMAIN.

I do not know what you are streaming, but any videos/audio files that are streamed on proxied DNS records will need a Stream subscription or else you will be violating our policies.