Need help confirming Google Bot traffic

I’m fairly new to analyzing CF’s traffic to confirm good versus bad traffic, but have been doing a good deal of reading on spoofing Google’s Bots to make it hard to filter them.

Yesterday (into today) we’ve been seeing a ton of traffic to our site with the User Agent “AdsBot-Google (+”, ASN of “15169”, which from my research are both associated with Google so I hesitate to block it. Thing is, according to CF some of the individual files (e.g. css, js, etc.) are being hit several thousand times in a day (see screenshot, which shows the “crawler” activity on the noted files just within the last 6 hours), which makes me believe this is NOT Google.

The crawler is also showing as coming from France (a known Google Bot origin) and using IP addresses in the 66.249.92.XX range. I did a reverse DNS lookup on a few and they all come back as being associated with Google. But I know these can be spoofed, right?

My questions are:

  1. Are there sure fire ways to tell if a visitor is a legit Google Bot using data in CF, and if so what do I look for outside of what I already have?
  2. Is it normal for a single file to be hit thousands of times a day as shown in the screenshot?
  3. If this is indeed NOT looking to be Google, what can I do to block this traffic without actually blocking legit GoogleBot?

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.