502: how to increase the 30 second timeout

Our servers create a 200 response and send that along but we are hitting a timeout at about 30 seconds. All the documentation suggests we’d get a 524 after 100 seconds, but that doesn’t seem to be the case.

This is a django app behind ssl, nginx, and gunicorn. We’ve checked all the relevant settings and have them set very high (600 seconds).

[2019-12-11 18:15:41,307] {/app/cwith/api1/middleware.py:51} WARNING - Request {'path': '/cwith/v0/test_sleep/45', 'ip_address': '0.0.0.0', 'method': 'GET', 'referrer': None, 'data': {'get': {}, 'post': {}}, 'timestamp': datetime.datetime(2019, 12, 11, 18, 14, 56, 242125, tzinfo=<UTC>)} Response 200 took=45065 ms

1 Like

Ok, this was finally resolved. This issue is that we use GCP / GCE / GKE (google cloud platform) and the load balancers (Kubernetes / k8s services and ingresses) have a vendor specific timeout setting which defaults to 30 seconds. (Note: This is probably an issue for AWS as well.)

https://cloud.google.com/kubernetes-engine/docs/how-to/configure-backend-service#creating_a_backendconfig

We had to create the special BackendConfig object like this:

`
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: long-timeout-backend
spec:
  # this is the most cloudflare allows:
  timeoutSec: 100
  connectionDraining:
    drainingTimeoutSec: 60
`

And include an annotation in our Service config like this:

`
apiVersion: v1                                                                                                                                                                                                                                                                                                                                                                             
kind: Service
metadata:
  name: test-api-staging-service
  labels:
    app: test-api
    role: staging
    client: all
  annotations:
    beta.cloud.google.com/backend-config: '{"ports": {"8000":"long-timeout-backend"}}'
spec:
  [...]
  type: LoadBalancer
  ports:
  - port: 8000
`
2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.