Some 413 responses are getting turned into 520s

Hello,
We’re using Cloudflare in front of a Google Cloud load balancer. We have a route on our website that users can use to upload avatars, and if an image is uploaded that is over a max size we have configured in nginx, nginx returns an HTTP 413 for the request. We’ve noticed, however, that when an image is uploaded that is over the nginx limit we have configured, sometimes we get back the HTTP 413 response we expect, and sometimes we get back an HTTP 520. Looking at our Google Cloud load balancer logs, the requests that are leading to 520s being returned from Cloudflare are returning 413s from Google Cloud. From examining the requests that result in 413s and the requests that result in 520s, they seem identical. Here’s an example below (slightly edited to remove some sensitive data):

This request resulted in an HTTP 413:

REQUEST

Request URL: https://staging.notability.com/avatar/upload
Request Method: POST
Status Code: 413 
Remote Address: [2606:4700:20::681a:e47]:443
Referrer Policy: strict-origin-when-cross-origin

RESPONSE HEADERS

cf-cache-status: DYNAMIC
cf-ray: 70297cd858eb97c9-SJC
content-type: text/html
date: Wed, 27 Apr 2022 18:05:42 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=uElsXMqO4v%2FWXxsNo5T6EsLvv91zwIn9L%2Fe9IwrQXyxjt5Lt46ZxMTbJ655m90hI2yYIA4OZ%2BAn3SEKaUo8YbHPJE%2BabhoV%2BIoVVtRVIQ56WlzbMQq1boSvzfQqyvyM5jzyEKkMrzL17vb0U7IwVu2Qt3NA%3D"}],"group":"cf-nel","max_age":604800}
server: cloudflare
via: 1.1 google

REQUEST HEADERS

:authority: staging.notability.com
:method: POST
:path: /avatar/upload
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
content-length: 6732514
content-type: multipart/form-data; boundary=----WebKitFormBoundaryPEzrFJvnBptOYvUb
cookie: nb-auth-token=xxx; _ga=xxx; _ga_BC058XCMF9=xxx
origin: https://staging.notability.com
referer: https://staging.notability.com/gallery/profile/christhechris
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36

This one resulted in a 520:

REQUEST

Request URL: https://staging.notability.com/avatar/upload
Request Method: POST
Status Code: 520 
Remote Address: [2606:4700:20::681a:e47]:443
Referrer Policy: strict-origin-when-cross-origin

RESPONSE HEADERS

cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
cf-ray: 70297f738e0397c9-SJC
content-type: text/html; charset=UTF-8
date: Wed, 27 Apr 2022 18:07:28 GMT
expires: Thu, 01 Jan 1970 00:00:01 GMT
referrer-policy: same-origin
server: cloudflare
set-cookie: cf_use_ob=0; path=/; expires=Wed, 27-Apr-22 18:07:58 GMT
x-frame-options: SAMEORIGIN

REQUEST HEADERS

:authority: staging.notability.com
:method: POST
:path: /avatar/upload
:scheme: https
accept: */*
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
content-length: 6732514
content-type: multipart/form-data; boundary=----WebKitFormBoundary91iZAyAFTfkAKw3w
cookie: nb-auth-token=xxx; _ga=xxx; _ga_BC058XCMF9=xxx
origin: https://staging.notability.com
referer: https://staging.notability.com/gallery/profile/christhechris
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"
sec-fetch-dest: empty
sec-fetch-mode: cors
sec-fetch-site: same-origin
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36

Here’s an example where I clicked on the upload button for the same image several times and you can see the responses are sometimes 413s and sometimes 520s:

And here’s a Google Cloud load balancer log for the 520 request I included above where you can see a 413 status code and a ‘response_sent_by_backend’ status:

I’ve also tried sending requests to nginx from within our Kubernetes cluster and directly to our ingress so that we bypass Cloudflare altogether, and I have received a 413 every time. I also tried changing our code to increase the nginx size limit and make our code return a 500 instead. AFAICT this fixes the 520 issue and is a successful workaround, but we’re still puzzled by what’s happening here.

So basically, tl;dr: how can I debug this and figure out what is going wrong?

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.