Expertini.com Job Site cannot be crawled by Moz, Ahref, Smerush

Website, Application, Performance
1

Dear Team Members,

We’re Expertini.Com a prominent job site and host over a million of jobs globally posted on our site.

We’re facing an issue that Moz - RogerBot is unable to crawl our site although we have added it in Security WAF Custom Rule as (http.user_agent eq “rogerbot”) in Skip Mode and for other bots also as follows:

(http.user_agent eq “User-Agent:IFTTT-Protocol/v1” or http.host eq “cdn.expertini.com” or http.user_agent eq “Rogerbot” or http.user_agent eq “rogerbot” or http.user_agent eq “Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)” or http.user_agent eq “SemrushBot” or http.user_agent eq “SiteAuditBot” or http.user_agent eq “SemrushBot-BA” or http.user_agent eq “SemrushBot-SI” or http.user_agent eq “SemrushBot-SWA” or http.user_agent eq “SemrushBot-CT” or http.user_agent eq “SplitSignalBot” or http.user_agent eq “SemrushBot-COUB” or http.user_agent eq “MJ12bot”)

Still Getting below notice from Moz

image
image1430×273 45.8 KB

Please suggest and guide. We need help to fix this issue

Thank You

3

According to Moz, the user agent string isn’t just “rogerbot”:

https://moz.com/help/moz-procedures/crawlers/rogerbot

It’s better to use a “contains” if you’re having difficulty matching the user agent string.

Also, their error message isn’t saying they can’t reach your site. It’s saying they’re prohibited by robots.txt or a robots tag. Best to double-check your firewall events log. It’s possible Moz just isn’t good at error reporting.

4

Thanks for your information. Just a few points our robots.txt file don’t block it. But from Cloudflare point of you if we enable the skip option with WAF for all bots which we want to crawl our site will be sufficient?

Thanks