Cannot allow naver crawler

Hi,
There is a problem that the naver.com crawler (yeti) cannot access only certain domains.
https://japanpang.com and https://sendmi.co.kr are the same server,
The firewall policy is the same because it is specified on a per-user basis only.
Sendmi can be accessed normally by Naver crawlers, but japanpang cannot.
I tried registering AS23576 with allow, but it still doesn’t work.
Please check.
thank you.

Do you see any activity in your Firewall Events Log that show Naver getting blocked or challenged?

1 Like

No, there are no logs.

When I turn off the proxy, the naver bot approaches well after a while.
These ips are these and all are AS23576.
110.93.150.220
114.111.32.106
114.111.32.196
125.209.235.170
125.209.235.186
211.249.46.159

turn on the proxy again, cannot access it.
There is nothing left in the firewall events log.

As a result of inquiring with Naver, they says ‘Connection timed out’.

It’s difficult to diagnose without seeing the content of the actual response, but that may be your server not responding to Cloudflare when it proxied Naver. That’s certainly a strange situation. Would you be able to get the exact Request header they’re sending?

1 Like

this is headers.
2021-05-24 22:24:19 114.111.32.45
GET / HTTP/1.1
Host: japanpang.com
Connection: Keep-Alive
User-Agent: Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/spd)
Accept-Encoding: gzip,deflate
Accept: */*
Accept-Language: ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4

This is an error message from Naver’s answer.
org.apache.http.conn.HttpHostConnectException: Connect to japanpang.com:443 [/172.67.156.226] failed: ?? ?? ?? (Connection timed out) at connect(DefaultHttpClientConnectionOperator.java:159) / at connect(SunnyHttpClientConnectionOperator.java:111) caused by java.net.ConnectException: ?? ?? ?? (Connection timed out) at socketConnect(Native Method) / at doConnect(AbstractPlainSocketImpl.java:350) / at connectSocket(SunnySSLConnectionSocketFactory.java:115)

thank you.

sendmi.co.kr is no problem.
japanpang.com is the only problem.
Why does this problem occur?

That’s certainly a timeout, though Cloudflare would normally respond with something to let them know there’s a timeout at your server.

Can you find out how long Naver waits before timing out? Cloudflare’s timeout is 100 seconds.

1 Like

Naver waits only 10 seconds.
Other domains on the same server work well and respond well with 100ms when tested with Chrome.

thank you.

1 Like

I tried setting it up in another account of Cloudflare for testing, but it was confirmed that Naver was not able to access the same.

Existing name servers
doug.ns.cloudflare.com
venus.ns.cloudflare.com
and the changed nameserver is
ben.ns.cloudflare.com
kira.ns.cloudflare.com

Firewall Events > Activity log > No firewall events found matching your filters

Could the bug be caused by the domain name?

After a few days of changing the nameserver, it was resolved.
It is confirmed that it works even after returning the nameservers to their original state.
I think the problem with Cloudflare is right,
It seems like a good way to try next time if the connection blocking problem of unknown cause occurs again.

Cloudflare has a lot of problems these days.
This time I get a 522 error.
If the server is fine and proxy is turned off, it works normally.

There is no communication between Cloudflare and servers in LA and Silicon Valley.

I’m not sure what that’s supposed to mean. My servers are in LA and they’re routing through LAX.

And here’s my Cloudflare tunnel list. The two blanks are decommissioned servers I’ve not yet removed tunnels for.
Screen Shot 2021-06-11 at 6.25.33 AM

1 Like

It seems that this problem was caused by something else.
It was solved.
thank you.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.