My friend who I debugged this with has opened a ticket so I’m sure Cloudflare will fix this soon, but in the meantime I wanted to post this here for anyone having this issue.
Do not enable HTTP/2 on your backend server. This causes sporadic 520 errors on the CF side. We found that sending an OPTIONS request would trigger the 520 about 100% of the time, but occasionally it would happen to other requests too. The backend logs show a stream of normal HTTP/1.1 requests and then randomly CF uses a HTTP/2 request, and the server responds correctly with HTTP/2 but Cloudflare shows a 520.