Hi everyone,
Thanks for everyone at Cloudflare team to bring out some nice products for free. You’re helping to build a better Internet (even if some critical issues about centralization are arised).
Today I’d like to talk about a recent issue I just discovered of Cloudflare Firewall. Cloudflare Firewall is a tool to protect your website from DoS/DDoS attacks and it can be configured to require additional verification to bots or some visitors from a country, or a particular User Agent. I discovered badly that, even if rate limiting was disabled, Cloudflare blocked by default all the good search engine bots (I’m referring in particular to Google Search Engine Crawlers, that are not bad actors nor attackers) with the reason “Rate limiting”.
This had negative impact of website since Google crawler thought it can not crawl my website, meaning my website was such “full” of visitors that my host can not serve any pages. Result? Negative impact on SEO. I just whitelisted all the bots of Google, Yahoo, Duckduckgo, Bing, and many others (based on ASN or IP given by those companies, to prevent fake ones) and now my website is trending pretty well on Search engines (without touching anything related to content).
I’d like to warn all the people that take care about SEO to avoid Cloudflare firewall, until the company starts to work with major actors for SEO. Cloudflare Team has partnerships with all over the globe, a partnership with Google or Web.dev would be very helpful for all the community.
Can you confirm that you don’t have rate-limiting enabled at all?
We have an escalation open for another post here with an unknown rate limit, not on search engine bots though. I can add yours to that and follow up on it.
I think that this request comes from a Cloudflare worker and not from Google itself. it’s a worker IP that could come from any person using workers, not necessarily you,
@cool_user_xmr do you happen to have any Cloudflare Workers running on your domain (You can check this in the Workers tab at the dashboard).
The rate-limit that you are currently running into is part of Cloudflare’s abuse protection that’s used for Workers.
Cloudflare’s abuse protection methods do not affect well-intentioned traffic. However, if you send many thousands of requests per second from a small number of client IP addresses, you can inadvertently trigger Cloudflare’s abuse protection. If you expect to receive 1015 errors in response to traffic or expect your application to incur these errors, contact Cloudflare to increase your limit.
To get this limitation lifted, you will have to contact Cloudflare’s support and ask them to increase your limit. You can refer to the link I posted above for that.
do you happen to have any Cloudflare Workers running on your domain (You can check this in the Workers tab at the dashboard).
I confirm I did not have any Cloudflare Workers running on my domain. This is why it looked strange to me.
I think that this request comes from a Cloudflare worker and not from Google itself. it’s a worker IP that could come from any person using workers, not necessarily you,
That’s weird then, as the worker firewall rule is only supposed to get triggered when Cloudflare Workers’s abuse protection kicks in.
Can you open a support ticket about your situation and then post the ticket ID here?
Looking at the user-agent there’s “PTST” which from googling is from WebPageTest
That’s interesting. Are those tests automatically executed or manually? That’s something I should ask to Google support. Anyway, it should not give any Rate limiting error.
By the way, how can I be sure this did not happen with any other bots as well? Or any other visitors?
It’s unlikely at least that this directly affected normal user traffic, as the worker abuse protection is only triggered for ip addresses that are sending thousands of requests per second, as mentioned in the small documentation quote I sent earlier.
I will further escalate this topic and ticket to Cloudflare’s support so they can take a look at what’s exactly happening here that’s causing you to run into the Workers abuse protection.
