Thanks for everyone at Cloudflare team to bring out some nice products for free. You’re helping to build a better Internet (even if some critical issues about centralization are arised).
Today I’d like to talk about a recent issue I just discovered of Cloudflare Firewall. Cloudflare Firewall is a tool to protect your website from DoS/DDoS attacks and it can be configured to require additional verification to bots or some visitors from a country, or a particular User Agent. I discovered badly that, even if rate limiting was disabled, Cloudflare blocked by default all the good search engine bots (I’m referring in particular to Google Search Engine Crawlers, that are not bad actors nor attackers) with the reason “Rate limiting”.
This had negative impact of website since Google crawler thought it can not crawl my website, meaning my website was such “full” of visitors that my host can not serve any pages. Result? Negative impact on SEO. I just whitelisted all the bots of Google, Yahoo, Duckduckgo, Bing, and many others (based on ASN or IP given by those companies, to prevent fake ones) and now my website is trending pretty well on Search engines (without touching anything related to content).
I’d like to warn all the people that take care about SEO to avoid Cloudflare firewall, until the company starts to work with major actors for SEO. Cloudflare Team has partnerships with all over the globe, a partnership with Google or Web.dev would be very helpful for all the community.