Hi Sid,
There’s very little anyone in this community can help in case of a 524 error other than to point you to our community tutorial on the subject, which you can find here: 524.
As far as having to turn APO off because it was caching everything, please make sure your APO installation follow the documented best-practices. APO is meant to cache everything (for up to 30 days), while revalidating and refreshing content within 30 seconds after it is changed.
APO is supposed to bypass requests under certain conditions. One of them is the presence of any query string, except for a few that are used mostly by marketing and analytics (utm_*, fbclid etc.). It is also supposed to bypass requests with certain cookies (please see Query parameters and cached responses · Cloudflare Automatic Platform Optimization docs for a list of cookie prefixes.)
If your developers set any of these cookies, or make use of a (non-marketing or analytics) query string to identify requests for logged-in content, these requests should not be cached. If that’s happening, please open a ticket and post the number here for further investigation.
When I visit my APO test site using a regular Chrome tab, I always get BYPASS (as I’m logged in). So I need to visit using incognito to see the cached response. Now, if your site has other user identification than the cookie set by WordPress, you’d need to talk to your developer about making sure Cloudflare APO has some element (cookie or query string) to make the decision not to cache.