Robots.txt forbidden

Website, Application, Performance DNS & Network
, ,
1

Hi,

I’m still having the same error as already mentionned in this topic (Robots.txt get forbidden from Hong Kong).
I’ve push a little bit the investigation and found something that could be interesting.
I’ve used this site (https://www.websitepulse.com/tools/china-firewall-test) and tried all the location from China (beijing, hong-kong, Guangzhou and Shanghai) and noticed that 3 of them get a 403 response code but Hong-kong. Inspecting the HTTP headers responses, I’ve also noticed that all 3 servers getting a 403, used a Cloudflare server (CF-RAY) like xxxxxxxxxxx-LAX where as the one used by Hong-Kong was xxxxxxxxx-HKG.
I’ve also (from the page above (websitepulse.com) notice that all other location get a 200 response code. Do you think the 403 returned code could be caused by the Cloudflare RAYID server (xxxxxx-LAX) ?
That’s the only thing I’ve found in common for all tested locations getting a 403 response code.

Thanks

Regards

2

I just tested from the same site and get 200 OK responses.

Tested From: Shanghai, China
Tested At: 2018-06-12
17:41:28 (GMT +00:00)
URL Tested: https://yourdomain.com/robots.txt
Resolved As: 104.20.34.52
Status: OK
Response Time: 1.330 sec
DNS: 0.149 sec
Connect: 0.153 sec
Redirect: 0.000 sec
First Byte: 0.686 sec
Last Byte: 0.343 sec
Size: 60949 bytes
HTTP Headers
HTTP/1.1 200 OK
Date: Tue, 12 Jun 2018 17:41:28 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=dfd79b8cd7b826c1b2a24a1afbf1860b21528825288;expires=Wed, 12-Jun-19 17:41:28
GMT;path=/;domain=.yourdomain.com;HttpOnly;Secure
Last-Modified: Thu, 07 Jun 2018 13:13:39 GMT
Referrer-Policy: no-referrer-when-downgrade
Strict-Transport-Security: max-age=31536000;includeSubDomains;preload
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1;mode=block
Cache-Control: public, max-age=604800
Content-Security-Policy: img-src * data:;script-src * data: https://www.google-analytics.com
https://www.google.com/recaptcha/api.js https://www.gstatic.com ajax.googleapis.com https://cdn.ampproject.org
‘unsafe-inline’ ‘unsafe-eval’
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: origin, x-requested-with, content-type, authorization
Access-Control-Allow-Methods: GET, POST, OPTIONS
CF-Cache-Status: HIT
Expires: Tue, 19 Jun 2018 17:41:28 GMT
Expect-CT: max-age=604800, report-uri=“https://report-uri.Cloudflare.com/cdn-cgi/beacon/expect-ct”
Server: Cloudflare
CF-RAY: 429e1bc40c1222c4-LAX

3

@cs-cf Hi thanks for your quick reply.
Actually, I think the problem has been solved by purging Cloudflare’s CDN cache. You probably checked it just before this action. So now for all failing proxy servers mentioned above, the status code is 200, has expected.
The only thing I don’t understand, is why we’ve getting a 403 because of a “unsynchronized” cache.
However, the problem seem to be solved, so all is ok for now.
Thanks anyway

4

Cloudflare will cache a 403 response for 5 minutes: https://support.cloudflare.com/hc/en-us/articles/202775670-How-Do-I-Tell-Cloudflare-What-to-Cache-

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.