How to Safely Allow the FeedFetcher-Google Bot

Website, Application, Performance Security
1

What is the name of the domain?

t4gsports.com

What is the issue you’re encountering

Cloudflare Blocking User Agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)

What steps have you taken to resolve the issue?

  1. Creating a Custom Rule allowing ASN 15169,
  2. Creating a Custom Rule allowing the user agent FeedFetcher-Google; (+http://www.google.com/feedfetcher.html) but that can be spoofed so I removed it.

I could add all of the IP addresses Google uses for the FeedFetcher bot, but that’s so much to manage, and I presume they can change.

I added a custom rule to skip all Super Bot Fighting Mode Rules for the Verified Bot Category - Feed Fetcher (among others), and it allows the Google Producer bot but denies the FeedFetcher-Google Bot. I also had it skip all managed rules.

I presume the FeedFetcher-Google bot is not recognized by Cloudflare as a verified bot. I am aware that it is not a well-known bot and doesn’t follow some of the standard crawler procedures.

Was the site working with SSL prior to adding it to Cloudflare?

Yes

What is the current SSL/TLS setting?

Full (strict)

2

I will leave the ASN rule up since that ASN is safe from my research and talking with SiteGround.

3

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.