WAF rules and 403 forbidden Google bot

Hi,

My website is getting scraped and costing me money. I created a URI full WAF rule with a JS challenge. This has stopped the scraping, but now some of my pages are getting removed from search engines.

In search console, I get a 403 forbidden error for some URLs but not all.

I’ve been in touch with Kinsta, and the issue is not related to the server.

When i switch off the rule, Google can crawl the page again. I looked at keeping the rule in place but switching off the ‘bad robot’ feature within managed rules. That seemed to create the 403 error again.

My developer isn’t available again for a few days so I’m trying to find a fix.

It’s a frustrating situation as I don’t have the knowledge or understanding to cure the issue. I need to prevent scraping site reports but allow Google bots to crawl.

I have read various threads in the community but nothing specific to 403 errors.

I would appreciate some instructions or suggestions please.

Regards
Marcus

Can you share your rule? We can help let you know the modifications needed.
You will need to allow verified bots.

Thank you for responding. Here is the rule attached. Scrapping has stopped. But I had to disable this and allow google bot to crawl the pages:

Regards
Marcus

You mean, this custom rule is JS challenging google bot and that causes 403 error?
In that case, you might want to create another custom rule to allow google bot.
Can you try to create Skip rule for User agent contains google or googlebot?
And please make sure to place higher order than the rule causing 403 error.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.