WAF blocking google bot?


#1

Hi there, Little help.

Does google bot gets blocked when we turned On the WAF? I have seen a lot of reports in my firewall event log. It’s happening all around the world like Russia, UK etc.

So It going to be a problem If I leave it alone.

I have a tool called screamingFrog which we can emulate the google bot user agent and crawl the website.

Here are the results.

For debugging the event raw log,

This code I emulated the google bot crawling with the screamingfrog software https://cl.ly/4c08205c711c

And this is the previous report I saw https://cl.ly/a155c1e4f144

I’m using the WordPress blog.

Any help is highly appreciated.


#2

Generally no, WAF won’t block the actual GoogleBot bot. It will tend to block bots pretending to be GoogleBot though (abusers know that GoogleBot is often whitelisted and therefore will pretend to be GoogleBot to take advantage of the same).


#3

Aha! So technically it stops spammers, or scrapper who visit the site by emulating as Googlebot.

I will set up a log analysis tool for this to get a clear idea.

Thanks, Dave.


#4

I was looking at my logs, interestingly it even overrides country based whitelisting (which is probably good, but a little surprising).


closed #5

This topic was automatically closed after 14 days. New replies are no longer allowed.