I should have mentioned, that you also need to wipe all possible caches (also Cloudflare caches - Purge Everything) and regenerate the files generated by autooptimize.
Now I just see 404 errors, no 524 and no 500 errors anymore.