Does Cloudflare gRPC have a timeout?

[miroberto]

Hello,
we use Kubernetes (GKE) on GCP and our main gateway is controlled by Istio.
We have two Golang applications (client - server) that communicate via gRPC.
We have enabled the gPRC check on our domain on Cloudflare.
Everything works fine, but during a gPRC server side streaming, sometime we have long time (a few minutes) without any message and we get this error:

error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR

This is our client, waiting for new messages:

for {
	status, err := stream.Recv()
	if err == io.EOF {
		break
	}
	if err != nil {
		log.Fatalf("error: %v", err)
	}
	fmt.Printf("%s", status.Message)
}

I’ve already tried to add a keep alive client side:

var kacp = keepalive.ClientParameters{
	Time:                10 * time.Second,
	Timeout:             time.Second,
	PermitWithoutStream: true,
}

Without Cloudflare, our applications work fine. Do you know if Cloudflare have some kind of timeout on the gRPC streaming? I’ve noticed the error appears always after 1m 30s / 1m 40s since last message.

If you need any further details, I’ll be glad to help.
Thanks

[miroberto]

We’ve also opened a support ticket, here’s the reply:

The timings here is interesting 1m 40s - Cloudflare has a timeout for our proxy for 524 HTTP/S responses that happens at 100 seconds, it is a HTTP timeout if we do not see any data transfer for 100 seconds we close the HTTP request and serve a 524 - I suspect this is the same proxy timeout your experiencing here.

If your server is processing for longer than 100 seconds and not passing any data back to the client, that could explain this.

Is it possible to implement a keep-alive on the server, returning a keep-alive back to the client?

We have implemented a keep-alive on server-side and we don’t face the problem anymore.
Thanks