How open CF for Mail.ru Search Bot?

crdpro · January 22, 2021, 12:11am

Hello.
Mail.ru Search Bot can not access to my site.

User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0;
Error:
HTTP/1.1 403 Forbidden
Date: Thu, 21 Jan 2021 23:27:51 GMT
Content-Type: text/plain; charset=UTF-8
Content-Length: 16
Connection: keep-alive
Set-Cookie: __cfduid=d865ccb0cb0df483c91599ca612c245bb1611271671; expires=Sat, 20-Feb-21 23:27:51 GMT; path=/; domain=mydomainaddress; HttpOnly; SameSite=Lax; Secure
X-Frame-Options: SAMEORIGIN
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
cf-request-id: 07c8dfa52700003374ee3db000000001
Report-To: {“endpoints”:[{“url”:“https://a.nel.Cloudflare.com/report?s=1h9%2BTuY3KNdywkTP2HwT%2FuQhUMLPN7niIN0RfkYcQSZ%2Bxxp9o1z31oJGgEA6T6ThJsYQJzrPOjSQq0epR4fl7WTJDcb9v76eWCQ%3D”}],“group”:“cf-nel”,“max_age”:604800}
NEL: {“max_age”:604800,“report_to”:“cf-nel”}
Server: Cloudflare
CF-RAY: 6154cee83ce73374-DME

How I can fixed it?

sdayman · January 22, 2021, 12:13am

Take a look at your Firewall Event Log. It should be super easy to filter on the CF-RAY ID. That should show you what setting blocked access.

crdpro · January 22, 2021, 12:21am

Where and how I can look Firewall Event Log?

michael · January 22, 2021, 12:26am

On the Firewall tab of the dashboard. Click on Filter, select Ray ID and enter the Ray Id in the box.

crdpro · January 22, 2021, 12:37am

I add Ray ID 6154cee83ce73374-DME

Activity log

Edit columns
No firewall events found matching your filters

What can I do?

sdayman · January 22, 2021, 12:49am

Try without the -DME:

6154cee83ce73374

crdpro · January 22, 2021, 12:52am

Ray ID

6154cee83ce73374

Method

GET

HTTP Version

HTTP/1.1

Host

Path

/

Query string

Empty query string

User agent

User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0;

IP address

213.159.210.11

ASN

AS29182 THEFIRST-AS

Country

Russian Federation

Service

Browser integrity check

Action taken

Block

[Export event ](data:text/json;charset=utf-8,%7B%0A%20%20%22action%22%3A%20%22drop%22%2C%0A%20%20%22clientASNDescription%22%3A%20%22THEFIRST-AS%22%2C%0A%20%20%22clientAsn%22%3A%20%2229182%22%2C%0A%20%20%22clientCountryName%22%3A%20%22RU%22%2C%0A%20%20%22clientIP%22%3A%20%22213.159.210.11%22%2C%0A%20%20%22clientRequestHTTPHost%22%3A%20%22domain.com%22%2C%0A%20%20%22clientRequestHTTPMethodName%22%3A%20%22GET%22%2C%0A%20%20%22clientRequestHTTPProtocol%22%3A%20%22HTTP%2F1.1%22%2C%0A%20%20%22clientRequestPath%22%3A%20%22%2F%22%2C%0A%20%20%22clientRequestQuery%22%3A%20%22%22%2C%0A%20%20%22datetime%22%3A%20%222021-01-21T23%3A27%3A51Z%22%2C%0A%20%20%22rayName%22%3A%20%226154cee83ce73374%22%2C%0A%20%20%22ruleId%22%3A%20%22bic%22%2C%0A%20%20%22source%22%3A%20%22bic%22%2C%0A%20%20%22userAgent%22%3A%20%22User-Agent%3A%20Mozilla%2F5.0%20(compatible%3B%20Linux%20x86_64%3B%20Mail.RU_Bot%2F2.0%3B%22%2C%0A%20%20%22matchIndex%22%3A%200%2C%0A%20%20%22metadata%22%3A%20%5B%5D%2C%0A%20%20%22sampleInterval%22%3A%201%0A%7D)

How I can fixed it?

crdpro · January 22, 2021, 12:56am

22 Jan, 2021 03:27:51

Block

Russian Federation

213.159.210.11

Browser integrity check

Ray ID

6154cee83ce73374

Method

GET

FilterExclude

HTTP Version

HTTP/1.1

Host

Path

/

Query string

Empty query string

User agent

User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0;

IP address

213.159.210.11

ASN

AS29182 THEFIRST-AS

Country

Russian Federation

Service

Browser integrity check

Action taken

Block

sdayman · January 22, 2021, 12:58am

It failed the Browser Integrity Check.

If you’re sure you want to allow that through, because Firewall -> Settings has Browser Integrity Check toggled, you need to add a Firewall Rule to allow their IP address, or ASN, or Mail.RU User Agent through:

crdpro · January 22, 2021, 1:09am

I do it but not work.
Robot can not access to site.

crdpro · January 22, 2021, 1:26am

22 Jan, 2021 05:24:19

Bypass

Russian Federation

213.159.210.11

Firewall rules

Ray ID

615579864c983aa1

Method

HEAD

HTTP Version

HTTP/1.1

FilterExclude

Host

Path

/

Query string

Empty query string

User agent

User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0;

IP address

213.159.210.11

ASN

AS29182 THEFIRST-AS

Country

Russian Federation

Service

Firewall rules

Rule ID

c324f7ca6b694166beca56c83ee83782

Rule name

Expression

(http.user_agent contains "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0;")

Action taken

Bypass

Services Bypassed

Browser integrity check

crdpro · January 22, 2021, 1:35am

How I can add ASN AS29182 THEFIRST-AS
Maybe will work?

sdayman · January 22, 2021, 1:53am

Change the User Agent dropdown to AS Number instead and use 29182

crdpro · January 22, 2021, 11:23am

How can i do it? Where add number?

sdayman · January 22, 2021, 2:00pm

Try this:

system · February 21, 2021, 1:58am

This topic was automatically closed after 30 days. New replies are no longer allowed.

Topic		Replies	Views
Getting 403 forbidden in my good bot (link checker) Security	8	4178	January 31, 2022
How do I always allow a known bot client to index my site Security ExpertReply	2	834	October 19, 2023
Cf.client.bot is matching all browsers Security	11	4380	April 19, 2022
Googlebot get 403 Security Center	4	2178	January 31, 2023
Blocking Google bot General	15	7134	July 12, 2020

How open CF for Mail.ru Search Bot?

Activity log

Related topics