we were troubleshooting a service downtime happened on 2022-08-11 between ariound 06:00:00 and 10:00:00 GMT
the issue was caused by many clients receving 403 errors on a particular POST request of our application.
after looking at our logs, we found out no request was actually forwarded to the application, so we finally pinpointed the issue, it was Cloudflare returning 403 errors due to large bodies, rule 100048.
the application is indeed proxied and behind Cloudflare WAF, but that rule is normally (and currently) disabled by default. we don’t have custom rulesets and only some of the provided rulesets are enabled without any overrides.
the request is normally large because of its payload, so it was a false positive.
we need to understand why that rule was automatically or magically enabled for that 3 hour time span, and understand if we have to add an override to disable it and some other to avoid “default - disabled” rules magically enabling themselves.
any clue on why it could have happened? were there temporary modifications to the default ruleset, rolledback after 3 hours?