Hi,
I have created a firewall rule to block Semrush Bot, but they are still getting through? Why is the firewall rule only partially working?
Semrush Bot Firewall Rule:
Field = UserAgent
Operator = Contains
Value = Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Expression =
(http.user_agent contains “Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)”)
Action = Block
I’d change “Contains” to just look for emrush (yes, I did leave the S off so it’s an easier match).
Hi,
Not sure I follow the logic here?
Well I tried your suggestion and know its made it worse? More Semrush traffic getting through?
Can you please post the actual Expression from your firewall rule?
Also, do you have any other Firewall Rules?
1 Like
(http.user_agent contains “Mozilla/5.0 (compatible; emrushBot/7~bl; +http://www.semrush.com/bot.html)”)
Yes,
R1
(http.user_agent contains “Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.c”)
R2
(http.user_agent contains “Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; [email protected]”)
That’s not what I said to do. Try:
(http.user_agent contains "emrush")
If you’re overly specific, it’s less likely to catch a bot due to slight deviations in the User Agent string.
Same goes with dotbot:
(http.user_agent contains "dotbot")
1 Like
Thank you, will try ;o)
1 Like
Hi,
Tried, but Semrush still getting through?
GuestIP: 185.191.171.35 » Whois
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html) Viewing user control panel Fri Jul 16, 2021 2:48 pm
no worries, is it worth adding an additional rule to block SEMRUSH ASN?
AS209366 SEMRUSH-AS
You can certainly add that to the rule. Let us know if that fixes it.
It looks like Semrush have two ASN`s, namely
% Information related to ‘185.191.171.0/24AS209366’
route: 185.191.171.0/24
origin: AS209366
mnt-by: MNT-DGGB
mnt-by: ADVANCEDHOSTERS-MNT
mnt-by: MNT-DATAWEBGLOBAL
created: 2021-06-24T10:03:14Z
last-modified: 2021-06-24T10:03:14Z
source: RIPE
ASN AS209366
Organisation SEMrush CY LTD
AS Name SEMRUSH-AS
IPv4 addresses announced 1536
IPv4 prefixes announced 5
Bogon prefixes announced 0
IPv4 space rank #26,782 out of 71,785
Registry RIPE
Registered Country CY
Registration Date 2019-02-21
Registration Change 2019-02-22
and
% Information related to ‘185.191.171.0/24AS396982’
route: 185.191.171.0/24
origin: AS396982
mnt-by: MNT-DGGB
mnt-by: ADVANCEDHOSTERS-MNT
mnt-by: MNT-DATAWEBGLOBAL
created: 2021-06-24T09:55:11Z
last-modified: 2021-06-24T09:55:11Z
source: RIPE
ASN AS396982
Organisation Google LLC
AS Name GOOGLE-PRIVATE-CLOUD
IPv4 addresses announced 487424
IPv4 prefixes announced 159
Bogon prefixes announced 0
IPv4 space rank #610 out of 71,785
Registry ARIN
Registered Country US
Registration Date 2018-08-16
Registration Change 2018-08-16
I’ve added AS209366 to Cloudflare FWR and is working, but traffic from AS396982 persists.
Not sure that I can create a Cloudflare FWR for AS396982 as this is a Google Private Cloud and not
SEMRUSH-AS. I dont want to be blocking legitimate traffic.
Traffic that is getting through is originating from AS396982.
ok,
the specific ip address range getting through is
- 185.191.171.0 - 185.191.171.255
so I have tried to create the following rule but its not accepted?
(ip.src eq 185.191.171.*)
*** Update ***
Think now resolved with these three rules…
Semrush IP Range
(ip.src in {185.191.171.0/24})
Semrush ASN
(ip.geoip.asnum eq 209366)
Semrush User Agent
(http.user_agent contains "emrush")
1 Like
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.