There is a problem that the naver.com crawler (yeti) cannot access only certain domains.
https://japanpang.com and https://sendmi.co.kr are the same server,
The firewall policy is the same because it is specified on a per-user basis only.
Sendmi can be accessed normally by Naver crawlers, but japanpang cannot.
I tried registering AS23576 with allow, but it still doesn’t work.
Do you see any activity in your Firewall Events Log that show Naver getting blocked or challenged?
No, there are no logs.
When I turn off the proxy, the naver bot approaches well after a while.
These ips are these and all are AS23576.
turn on the proxy again, cannot access it.
There is nothing left in the firewall events log.
As a result of inquiring with Naver, they says ‘Connection timed out’.
It’s difficult to diagnose without seeing the content of the actual response, but that may be your server not responding to Cloudflare when it proxied Naver. That’s certainly a strange situation. Would you be able to get the exact Request header they’re sending?
this is headers.
2021-05-24 22:24:19 220.127.116.11
GET / HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/spd)
This is an error message from Naver’s answer.
org.apache.http.conn.HttpHostConnectException: Connect to japanpang.com:443 [/18.104.22.168] failed: ?? ?? ?? (Connection timed out) at connect(DefaultHttpClientConnectionOperator.java:159) / at connect(SunnyHttpClientConnectionOperator.java:111) caused by java.net.ConnectException: ?? ?? ?? (Connection timed out) at socketConnect(Native Method) / at doConnect(AbstractPlainSocketImpl.java:350) / at connectSocket(SunnySSLConnectionSocketFactory.java:115)
sendmi.co.kr is no problem.
japanpang.com is the only problem.
Why does this problem occur?
That’s certainly a timeout, though Cloudflare would normally respond with something to let them know there’s a timeout at your server.
Can you find out how long Naver waits before timing out? Cloudflare’s timeout is 100 seconds.
Naver waits only 10 seconds.
Other domains on the same server work well and respond well with 100ms when tested with Chrome.
I tried setting it up in another account of Cloudflare for testing, but it was confirmed that Naver was not able to access the same.
Firewall Events > Activity log > No firewall events found matching your filters
Could the bug be caused by the domain name?
After a few days of changing the nameserver, it was resolved.
It is confirmed that it works even after returning the nameservers to their original state.
I think the problem with Cloudflare is right,
It seems like a good way to try next time if the connection blocking problem of unknown cause occurs again.
Cloudflare has a lot of problems these days.
This time I get a 522 error.
If the server is fine and proxy is turned off, it works normally.
There is no communication between Cloudflare and servers in LA and Silicon Valley.
I’m not sure what that’s supposed to mean. My servers are in LA and they’re routing through LAX.
And here’s my Cloudflare tunnel list. The two blanks are decommissioned servers I’ve not yet removed tunnels for.
It seems that this problem was caused by something else.
It was solved.
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.