Robots.txt get forbidden from Hong Kong



We’re facing a strange problem when we are trying to reach our URI ‘/robots.txt’.
When we’re doing this GET request from Singapour, we’ve got a 200 response code.
When we’re doing the same GET request from a server located in Hong Kong, we’ve got a 403 response. This behaviour is only observable from a sever from Hong Kong.
If we do the same request with another file, let’s say ‘robots_test.txt’, this works as expected, we 've got a 200 response code from Hong Kong server.
Is there anyone encountered the same strange behaviour/problem?

Thanks for any clue


Robots.txt forbidden

It could be the IP reputation of the IP in use is triggering the web application firewall or other settings in your account such as country code restrictions are blocking your test if these aerrors are Cloudflare branded. If they aren’t, then they are being returned from the origin.


@cscharff Hi, thanks for your prompt reply.
Actually the strange thing is that we do not even see this request in our apache logs from the targeted server where as we clearly see the same request from Singapour logged.
Looks like the request does not even reach our targeted server (the one holding our robots.txt file). Very strange
What do you think?


Do you see events on the firewall tab associated with the request?


@cscharff Hi, not at all, I don’t see any thing in the firewall tab passing. As said earlier, I don’t even see anything passing in my apache logs access logs nor in error logs. But I get a 503 error response page from (see picture in my PM)


Since the 403 doesn’t include Cloudflare branding this would appear to be an issue at your origin. Do you see this connection request in the webserver logs or in the logs of your firewall?


I don’t see any connection request in my webserver logs or in my firewall. Looks like the request is stopped before it reaches my origin webserver (behind cloudflare).

closed #8

This topic was automatically closed after 14 days. New replies are no longer allowed.