I solved it. Here’s the solution for anyone facing the same issue:
TL;DR: Add protocol: quic to your config.yml and restart cloudflared.
After reading the documentation I realized I was missing adding quic as the default transportation for cloudflared. Checking the logs it looks like cloudflared did try quic as the default transportation but it was somehow switching to http2:
{"level":"info","connIndex":0,"time":"2022-06-25T03:13:50Z","message":"Retrying connection in up to 32s seconds"}
{"level":"info","connIndex":0,"time":"2022-06-25T03:14:13Z","message":"Switching to fallback protocol http2"}
{"level":"debug","connIndex":0,"time":"2022-06-25T03:14:13Z","message":"Connecting via http2"}
So by adding protocol: quic and restarting the cloudflared UDP protocol is back working between WARP and my network segments tunneled through cloudflared.