Random cache error with 502 while the backend is infact available

Website, Application, Performance
1

What is the name of the domain?

maps.exponent.ch

What is the issue you’re encountering

We have defined the following ruleset to cache specific binary files. It works well for a few months and around last fall, some 502 errors started to appear. It works without issues when disable the Cloudflare cache/tunnel, and only appears on some of the Cloudflare nodes (e.g. using a VPN to get cache elsewhere works, cleaning the cache also temporarily fixes the issue). It is likely that the issue appears when the underlying data is changed. Ruleset for http_request_cache_settings with cache=true & respect_strong_etags=true: ((ends_with(http.request.uri.path, “.mvt”) or ends_with(http.request.uri.path, “.pbf”)) and http.host eq “SUB.DOMAIN.TLD”)

What are the steps to reproduce the issue?

curl https:// SUB.DOMAIN.TLD /planet/12/2117/1453.mvt -v

  • Host SUB.DOMAIN.TLD:443 was resolved.
  • IPv6: 2606:4700:3030::6815:1001, 2606:4700:3030::6815:5001, 2606:4700:3030::6815:4001, 2606:4700:3030::6815:6001, 2606:4700:3030::6815:7001, 2606:4700:3030::6815:2001, 2606:4700:3030::6815:3001
  • IPv4: 104.21.64.1, 104.21.48.1, 104.21.96.1, 104.21.80.1, 104.21.112.1, 104.21.16.1, 104.21.32.1
  • Trying 104.21.64.1:443…
  • Connected to SUB.DOMAIN.TLD (104.21.64.1) port 443
  • ALPN: curl offers h2,http/1.1
  • (304) (OUT), TLS handshake, Client hello (1):
  • CAfile: /etc/ssl/cert.pem
  • CApath: none
  • (304) (IN), TLS handshake, Server hello (2):
  • (304) (IN), TLS handshake, Unknown (8):
  • (304) (IN), TLS handshake, Certificate (11):
  • (304) (IN), TLS handshake, CERT verify (15):
  • (304) (IN), TLS handshake, Finished (20):
  • (304) (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
  • ALPN: server accepted h2
  • Server certificate:
  • subject: CN=DOMAIN.TLD
  • start date: Mar 8 16:30:26 2025 GMT
  • expire date: Jun 6 17:30:22 2025 GMT
  • subjectAltName: host “SUB.DOMAIN.TLD” matched cert’s “*.DOMAIN.TLD”
  • issuer: C=US; O=Google Trust Services; CN=WE1
  • SSL certificate verify ok.
  • using HTTP/2
  • [HTTP/2] [1] OPENED stream for https:// SUB.DOMAIN.TLD /planet/12/2117/1453.mvt
  • [HTTP/2] [1] [:method: GET]
  • [HTTP/2] [1] [:scheme: https]
  • [HTTP/2] [1] [:authority: SUB.DOMAIN.TLD]
  • [HTTP/2] [1] [:path: /planet/12/2117/1453.mvt]
  • [HTTP/2] [1] [user-agent: curl/8.7.1]
  • [HTTP/2] [1] [accept: /]

GET /planet/12/2117/1453.mvt HTTP/2
Host: SUB.DOMAIN.TLD
User-Agent: curl/8.7.1
Accept: /

  • Request completely sent off
    < HTTP/2 502
    < date: Mon, 10 Mar 2025 17:24:21 GMT
    < content-type: text/plain; charset=UTF-8
    < content-length: 15
    < report-to: {“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report/v4?s=1%2BtDbre%2BIrVrZM96pHol7B1V5RkQQ%2FX%2FuO%2FI1uSZQaytjV27W%2Bkp78LrZB0dYin%2F3c4mlqg2ubVT1D6sSb2VmQUtQv0D7CR1Rn%2F95dkytv3cJP5Xjxh0KBFyDzqxbRBq9Gki”}],“group”:“cf-nel”,“max_age”:604800}
    < nel: {“success_fraction”:0,“report_to”:“cf-nel”,“max_age”:604800}
    < x-frame-options: SAMEORIGIN
    < referrer-policy: same-origin
    < cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
    < expires: Thu, 01 Jan 1970 00:00:01 GMT
    < server: cloudflare
    < cf-ray: 91e4815139bd0071-CDG
    < alt-svc: h3=“:443”; ma=86400
    < server-timing: cfL4;desc=“?proto=TCP&rtt=162362&min_rtt=162251&rtt_var=61067&sent=6&recv=9&lost=0&retrans=0&sent_bytes=2890&recv_bytes=557&delivery_rate=22581&cwnd=241&unsent_bytes=0&cid=f9b8a11882861edd&ts=249&x=0”
    <
  • Connection #0 to host SUB.DOMAIN.TLD left intact
    error code: 502

curl --insecure https:// SERVICE_IP /planet/12/2117/1452.mvt -H “Host: SUB.DOMAIN.TLD” -v

  • Trying SERVICE_IP:443…
  • Connected to SERVICE_IP (SERVICE_IP) port 443
  • ALPN: curl offers h2,http/1.1
  • (304) (OUT), TLS handshake, Client hello (1):
  • (304) (IN), TLS handshake, Server hello (2):
  • (304) (IN), TLS handshake, Unknown (8):
  • (304) (IN), TLS handshake, Certificate (11):
  • (304) (IN), TLS handshake, CERT verify (15):
  • (304) (IN), TLS handshake, Finished (20):
  • (304) (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384 / [blank] / UNDEF
  • ALPN: server accepted h2
  • Server certificate:
  • subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  • start date: Feb 13 23:29:56 2025 GMT
  • expire date: Feb 13 23:29:56 2026 GMT
  • issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • using HTTP/2
  • [HTTP/2] [1] OPENED stream for https:// SERVICE_IP /planet/12/2117/1452.mvt
  • [HTTP/2] [1] [:method: GET]
  • [HTTP/2] [1] [:scheme: https]
  • [HTTP/2] [1] [:authority: SUB.DOMAIN.TLD]
  • [HTTP/2] [1] [:path: /planet/12/2117/1452.mvt]
  • [HTTP/2] [1] [user-agent: curl/8.7.1]
  • [HTTP/2] [1] [accept: /]

GET /planet/12/2117/1452.mvt HTTP/2
Host: SUB.DOMAIN.TLD
User-Agent: curl/8.7.1
Accept: /

  • Request completely sent off
    < HTTP/2 200
    < date: Mon, 10 Mar 2025 17:24:30 GMT
    < content-type: application/x-protobuf
    < accept-ranges: bytes
    < content-encoding: gzip
    < etag: “f419241ddcf9ad97”
    < strict-transport-security: max-age=31536000; includeSubDomains
    < cache-control: public, max-age=604800, s-maxage=259200, stale-while-revalidate=86400, stale-if-error=86400
    < vary: Origin
    <
    Warning: Binary output can mess up your terminal. Use “–output -” to tell
    Warning: curl to output it to your terminal anyway, or consider “–output
    Warning: ” to save to a file.
  • Failure writing output to destination, passed 2596 returned 4294967295
  • process_pending_input: nghttp2_session_mem_recv() returned -902:The user callback function failed
  • Connection #0 to host SERVICE_IP left intact
2

Hi there,

A 502 is not a caching error.
Here’s some info I gathered form the Ray ID provided

  "_source": {
    "timestamp": "2025-03-10T17:24:21Z",
x

    "message": " context: while reading response headers",
    "host": "maps.exponent.ch",
    "request": "GET /planet/12/2117/1453.mvt",
    "id": "91e4815139bd0071",
    "error": {
      "source": "Upstream",
      "type": "ConnectionClosed",
x
    },

Because of the message, I would guess that the response headers where not correctly formatted, where too large, or the origin does not fully support multiplexing.

So I would check the following:

  • Make sure the response headers do not exceed 16 KB.
  • Make sure the response has HTTP status code and response body.
  • Make sure your origin supports multiplexing, if not disable HTTP/2 to origin.
  • Make sure Cloudflare IP addresses are allowed unmitigated access to the origin. (Allowlist Cloudflare IP addresses)

Take care.