Cloudflare blocks Google (WebPage?) bots even if rate limiting was disabled

Hi everyone,
Thanks for everyone at Cloudflare team to bring out some nice products for free. You’re helping to build a better Internet (even if some critical issues about centralization are arised).

Today I’d like to talk about a recent issue I just discovered of Cloudflare Firewall. Cloudflare Firewall is a tool to protect your website from DoS/DDoS attacks and it can be configured to require additional verification to bots or some visitors from a country, or a particular User Agent. I discovered badly that, even if rate limiting was disabled, Cloudflare blocked by default all the good search engine bots (I’m referring in particular to Google Search Engine Crawlers, that are not bad actors nor attackers) with the reason “Rate limiting”.

This had negative impact of website since Google crawler thought it can not crawl my website, meaning my website was such “full” of visitors that my host can not serve any pages. Result? Negative impact on SEO. I just whitelisted all the bots of Google, Yahoo, Duckduckgo, Bing, and many others (based on ASN or IP given by those companies, to prevent fake ones) and now my website is trending pretty well on Search engines (without touching anything related to content).

I’d like to warn all the people that take care about SEO to avoid Cloudflare firewall, until the company starts to work with major actors for SEO. Cloudflare Team has partnerships with all over the globe, a partnership with Google or Web.dev would be very helpful for all the community.

1 Like

Where is the evidence of the rate limit or blocks you are referring to? I have never experienced this issue.

.
A lot of similar events (at least 40) were reported in the firewall. Available to share more details with Cloudflare team.

Hi @cool_user_xmr,

Can you confirm that you don’t have rate-limiting enabled at all?

We have an escalation open for another post here with an unknown rate limit, not on search engine bots though. I can add yours to that and follow up on it.

1 Like

It’s disabled. (I changed the language so other people can understand it)

1 Like

Thank you, I’ll add your issue to the existing escalation and follow up on it.

2 Likes

I’d like to help if this can improve Cloudflare for customers.

I think that this request comes from a Cloudflare worker and not from Google itself. it’s a worker IP that could come from any person using workers, not necessarily you,

1 Like

@cool_user_xmr do you happen to have any Cloudflare Workers running on your domain (You can check this in the Workers tab at the dashboard).

The rate-limit that you are currently running into is part of Cloudflare’s abuse protection that’s used for Workers.

https://developers.cloudflare.com/workers/platform/limits#request

Cloudflare’s abuse protection methods do not affect well-intentioned traffic. However, if you send many thousands of requests per second from a small number of client IP addresses, you can inadvertently trigger Cloudflare’s abuse protection. If you expect to receive 1015 errors in response to traffic or expect your application to incur these errors, contact Cloudflare to increase your limit.

To get this limitation lifted, you will have to contact Cloudflare’s support and ask them to increase your limit. You can refer to the link I posted above for that.

1 Like

do you happen to have any Cloudflare Workers running on your domain (You can check this in the Workers tab at the dashboard).

I confirm I did not have any Cloudflare Workers running on my domain. This is why it looked strange to me.

I think that this request comes from a Cloudflare worker and not from Google itself. it’s a worker IP that could come from any person using workers, not necessarily you,

Are you sure? From Google ASN?

Hm that is odd. Do you maybe then use APO?

https://developers.cloudflare.com/automatic-platform-optimization/

Nope. It’s not a Wordpress-based website, thus I did not enable APO.

That’s weird then, as the worker firewall rule is only supposed to get triggered when Cloudflare Workers’s abuse protection kicks in.
Can you open a support ticket about your situation and then post the ticket ID here?

Yes, I’m guessing that there is an error on MaxMind database (which is the source from which Cloudflare gets geo-location and information of IPs).

Either way, the user-agent shows that the agent is not google bot.

Looking at the user-agent there’s “PTST” which from googling is from WebPageTest

Let me open a support ticket, then. (#2219090)

Looking at the user-agent there’s “PTST” which from googling is from WebPageTest

That’s interesting. Are those tests automatically executed or manually? That’s something I should ask to Google support. Anyway, it should not give any Rate limiting error.

By the way, how can I be sure this did not happen with any other bots as well? Or any other visitors?

It’s unlikely at least that this directly affected normal user traffic, as the worker abuse protection is only triggered for ip addresses that are sending thousands of requests per second, as mentioned in the small documentation quote I sent earlier.

I will further escalate this topic and ticket to Cloudflare’s support so they can take a look at what’s exactly happening here that’s causing you to run into the Workers abuse protection.

1 Like

Thank you. I do really appreciate.

I just checkd my Cloudflare Firewall Logs and see the Same Problem.

Have over 580 Reports about Rate Limiting Google.

This is Very Bad.

In the Reports The Rule responsible for Blocking is named as “Unavailble”
Which is not one that is availble to be deactivated in this case.

So Cloudlfare does Block Google even when this should not have happened !!!

I think this Comes From Google PageSpeed Test.

They Test how much Load can the Website take
to avoid display a Weak Website in the Results i think.

This Rate Limiting should not have Happened !!!