I have some questions about the Cloudflare DDoS protection.
We suspect that we are sometimes triggering the DDoS protection of external sites. We have a service that downloads images from various sites, and sometimes we get an HTTP 503 response code along with the “Just a moment…” HTML page.
Many of these sites are our customers, who are explicitly allowing our use of their images.
We are wondering if there is something we can do to avoid this, wait more between requests, or something else? There does not seem to be any type of header that tells us to do rate limiting, as far as I can see.
We are also wondering if this is something site specific, or some sort of global block from Cloudflare. Do we have to contact each site individually for troubleshooting?
We do not know if all Cloudflare protected sites are affected, or only some of them.
It depends on a variety of factors. For example, which security level you’ve set or what your actual set up looks like. Is this check coming up if visitors access your site directly, or do other websites proxy the visitors traffic to your zone?
There is no check when visitors access our site directly. The problem occurs when we are downloading images from external sites. It is the software that seems to trigger the DDoS protection.
When I try to access these images manually through a browser, it works fine.
Could it be that our IP reputation is too low? How could we work on improving that?
Just to clarify, you try to access a third-party site that is behind Cloudflare and you are seeing this when downloading images?
In this case, the reason would be that the settings of the third-party sites have a stricter security level. Those checks are in place if a site wants to make sure that there are no automated requests taking place. In order to fix this, those sites would either need to whitelist you in their firewall rules, loosen their security level, or if you are a larger provider, we can also add you to a known bots list on request.
That is correct. Thank you, we will contact the respective sites. But we are also interested in being added as a known bot. How should we proceed with this?
Just create a support ticket with more detailed information and we’ll be able to forward your request to our bots team. But out of experience, this might take a while until it’s approved.