Random 520 Error only with Access policy applied

We are seeing strange behaviour with a single access policy protecting a single sub domain only.
We get 520 errors for site pages at a guess 25% of the time.
No upstream errors recorded and the origin servers are generating the pages in around 3 - 5 ms.
When the access policy is deleted (meaning there are no longer any access policies) the issues goes away.
Any clues to seeing what could be causing this?
Thanks a lot