My friend who I debugged this with has opened a ticket so I’m sure Cloudflare will fix this soon, but in the meantime I wanted to post this here for anyone having this issue.
Do not enable HTTP/2 on your backend server. This causes sporadic 520 errors on the CF side. We found that sending an
OPTIONS request would trigger the 520 about 100% of the time, but occasionally it would happen to other requests too. The backend logs show a stream of normal HTTP/1.1 requests and then randomly CF uses a HTTP/2 request, and the server responds correctly with HTTP/2 but Cloudflare shows a 520.
Looking into the CF docs there is a page that says that HTTP/2 is not used between origin and CF: Understanding Cloudflare HTTP/2 and HTTP/3 Support – Cloudflare Help Center
However, it seems this is an issue with CF since they are negotiating a HTTP/2 request in the first place.