I am receiving a 403 Forbidden when performing load tests at a rate of 12k req/s. When testing with less req/s (say 3k req/s) everything works fine. It seems I am hitting some internal limit. Is there a way to disable this safeguard?
What steps have you taken to resolve the issue?
I am also using Queues and KV inside the worker. To ensure the Queues are the issue, I created multiple queues and direct traffic to the queues at random, ensuring none of the queues hit their 5k ops/s limit. I am also using Worker caching for the results I get out of the KV so that I am not hitting that as much either.
So I believe the core issue is simply hitting an internal limit on Workers. I should also note that in the Cloudflare dashboard there are no reported errors in the logs (probably because my code didn’t even run), but under Analytics & Logs > Account Analytics I can see I have 165k 4xx errors. Any help/guidance on how to fix is appreciated!
I have done some more testing, and I can now confirm that this is Cloudflare blocking traffic. Logging a request I see the " Sorry, you have been blocked" page.
I am using Grafana K6 Cloud to do this testing, and from what I can see online this is done using a pool of IPs by default.
Is there a way to disable this firewall? I only need to take it down for few minutes so I can perform the tests. I feel like I have clicked on every tab in the dashboard and I cannot see any mention of workers being rate limited by Cloudflare.
The link I posted above tells you how to identify if the requests are triggering anit-abuse detections, and to raise a support ticket if you need limits lifting.
I took a look at the link you posted, but I think the post is outdated. It says “You can also confirm if you have been rate limited by anti-abuse Worker Rate Limiting by logging into the Cloudflare dashboard, selecting your account and zone, and going to Security > Events.” however, I have looked all over the dashboard and cannot see any Security tab anywhere. There is a “Security Center” with submenus “Security Insights”, “Infrastructure”, “Investigate”, and “Blocked Content (new)” but nothing like Events.
I don’t see a “Zone” mentioned anywhere, and when I click on your link I am taken to Account Home > Domains with the “Enter an existing domain” input focused. Is this trying to tell me that I need to setup a custom domain on my worker to view Zone Security settings? (Thank you for helping me! )
Ah ■■■■, yes, sorry, you are using workers.dev directly. Just go and raise an account support ticket to ask if the anti-abuse Workers Rate Limiting can be disabled for your account due to your load testing.
)