Robots.txt 301 redirect loop

Answer these questions to help the Community help you get started.

What is the domain name?
https://jaserodley.com

Have you searched for an answer?
Yes

Describe the issue you are having:
This site runs on WordPress, hosted on Cloudways. robots.txt is generated by a plugin called The SEO Framework and is not a static file.

When checking HTTP headers, a redirect loop is caused on this file (https://jaserodley.com/robots.txt). The output is below.

What error message or number are you receiving?

>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	1260339
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=sE0fMGGRdm%2B3KsT1%2FYh%2FCfszgk7QEc3mIr4iPCLIbqHvUPpymZHeFHnc9XxDBhYrWJyY07LzT4215l%2FBBYsdi%2Fvm2SViWjxSJBCl0e0oKQo65DtRXhqfSn4bB06QAn84IA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b2178b687024-CDG
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	42992
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Gq2%2B4TLJnlszYkcp907bAPomQMWFmvmn6mbsH1bsIKv9D77Fit58aqBpU1%2BhL%2F1atv52UDDEpofuFbIgh0oDC3YPtZPh6YvKXvLxsnjPZmecq7IvT73UwaTx1O%2BO%2FuoZMw%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b2180821008c-CDG
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	454037
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=9hnFLBOdyAothx1W1o7QstHIC2nPXJ180NxGyH4UMKmqFU9yaN0ZGKxz2ZykXotVhEMePtXUck3Ll8qII%2BU2jxbcZWiff111t5ilx9Bv9o2AijTugU3t%2BbW7MlzMD0NVgA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b2184b5fbbf7-FRA
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	169534
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Ykv9y48O4WSJapu8metqdCsbVzB7FuV0EuspYBaJHG7mpLSJN6P6kz1rhfaBUTZNzOp8JZ8e4Dkuv8bhJav2owB5V6hwYXya2vBvd1teuXmmTE4TZNLcrcI9BFdu5gu79Q%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b2189b9a0a50-AMS
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	1174611
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=BYuKGSs8e8%2BKWTcrjTbl8dnOW3YaeuY0AZalbUHQO6uz3OXG%2F3N6xqCw0sV3kVtnTO4rYQC%2F%2BkSkZ38zG9vKcWFaY1WZ1kBt4EdJXj%2Fa81eE54wzYxZPtTOnOO25V8lmXg%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b218dd691ca1-FRA
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	1238188
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=%2ByqkqgzXJmAUNePSeo36skZ6OJrD9vEdi%2BfibZw9mg6%2BcY9bVRBLiAf%2Bu6wBSeh5YxQaK%2FueHqDiXJgXf%2BHael7kcUweZBKjyb2C%2FSKHdBtkNe7%2B%2FGMVURxT1NVnR3LqYw%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b2193b9e3c9b-CDG
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	777930
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=GPYKdRJNZWM7yTqLrdvI6DyMAVsVU4cTcCOD%2FGZLTFGE584jW%2FqIoRgXGpUrheXJr9tt%2FcJkVFw4PnoOmBLKQxrSW5gb7Kl5JjE0GSesysICjBkNRnv5Sv2BeLyXW%2BJq0A%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b2196e8a37e0-FRA
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	169534
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=G1myGSwIofy%2BD%2FVSu131wQVPU7ys%2BA%2Fx09oobQqyY7vl2GRp2doEFIGrU%2Bhr7UREaMNqQfFtKGeKliYUg6RAa6ououJu3uSu4H805ZB9jOZcP%2FLerYDIXUIfKNUK%2F%2FJf%2Fw%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b219be9c0a78-AMS
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	169534
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=BIRXZAQ4PWoqOo27tZr9TRm2mFyrZ6YwF4qgdElmFPF8AXfB9TWKJL3G7TxjDwWPmbxgud%2BDRqKxsf66tn3EEBwmJqf6oyRbeLl20bMe2XBQidg23zmaNS9lIIrHl6h2TA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21a1efb0ae1-AMS
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	169534
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=HSyZ%2FvsNNXwVdehW0N3gxuizxaKGiTXRk7rqUH%2FxF%2B9YQ8qZe2rlQfNwzISlS%2BdsGwvU6SoawnmFs7%2FYzVS7ssfDquibEoh0%2FaUEMwrfh3M0PLmxKMfav3oHSztCy9mwAA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21a8f880bde-AMS
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	29794
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Fx2VNMy0B2UER3Npba2TRKtE9UykHoBRbS3cTMuh%2B7u%2Fwe0cr83v2YfrFJtsaKE0MGb7bxn%2B7TE145yYob5ixxsJyYA1eew6SVhMCV%2FfT0R4XIIm6nsvy%2BvPDHFvizm5dA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21acbed03d8-FRA
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	750330
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=EkmmY9i1x9pbd2hrjKox4KV8IuinVe5%2B%2BYJTJFiopN9hzMt%2BVp6eb7RpAuFnTWpT13SWgRs655aW7D%2FUdYuY3XQa04asZ36Wp5ONXB1fArXn5mGgQqPKsyUOJycttS9dkQ%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21b3f1863f8-LHR
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	42992
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=425SKnytC24%2FfpVULJBcer1Yi3F8cRAxN%2FIP%2FxDfQTD2bSZ8cFD%2BzJXXh46oS2YBIhVerGMDPz4qJNRWXGb1TPWJhJlW%2F4ZJkyodsGsyolPK5bbQWstXTIgShiR%2BVZ18CA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21baea6d52e-CDG
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	1197996
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=wgQU4jkWP1p1Zd6qM03zWlXGITtoGGKtiPpzHB%2F1FOqxy5tNRUGz%2FSvNV5Kt%2FAcv%2Fd3Y0FbJiICxiBjeufaBdhnuKOTOxflMYPIUqAn4vCBp4e88wgHCPAI2ncRt7ZpaOA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21c2d7ab930-AMS
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	483358
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=exFageisr8BYaIVpiWyFeZNO%2FcYmB844Sb6KBV%2FA58Avp7zT19GxyIgKH%2FOp3gSFQxYm5YeAuXFMCJ1aEQuPZgSAOKvmwhGI7z1CsXg7JELCcgfZMmFQIwjxrHVv0m3ytg%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21c8f14998c-CDG
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:24 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	1260339
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ffz5dJO1PJndh%2BJ8xKiwO8NxU%2F712y8LxshqPOYHknzgZvpsopnh80B6VQaSbXjEZAwB26sXnMr78nu9BsIAQflrjNrdsK4sGgdgGBPhly5I9vWHeMvWaqQa2vtio34FDA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21d0cef6f4c-CDG
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:25 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	525951
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Zx8KjabAmXQ%2BkVTN4F%2BvnE%2FrdXFNzQYU7ZO0lUEVX66F5KFS9duTDuSGVFkzB4xEuARckTOtEpoL6t6NjTYTaI5F%2Br95%2FHEeOhAtBKbPbnglfiB8Md0YZpDVKM%2FDgHwhHQ%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21d4f295d92-FRA
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:25 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	483359
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=k%2BbKyr19ZMGvVdRAqXyg%2B5DEe5N6Ql8DPpMuLtqBA6uy5CKxYzdfcPorWT6VRGDBf%2BZlv9WK%2BHEwgdLgxTgMrQTxTUvHIHy6C%2BWic6tOcbrTsKSII1YynZPQtDd9Mrf%2FXA%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21dbef80403-CDG
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:25 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	1197997
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=4n6eSz58p4BIehxpgepjRLn3P0Z4CqayPMwNapUWS14LjOxgDJbHlMUaiB5l281%2Ff%2FT7qqQisSp6d%2BIXE4eIIJv0XYRpbZLibtDwJ7cEHICVJqwKupsBSK%2FTEXqtSluFiw%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21e2f67660c-AMS
alt-svc:	h3=":443"; ma=86400




>>> https://jaserodley.com/robots.txt

> --------------------------------------------
> 301 Moved Permanently
> --------------------------------------------

Status:	301 Moved Permanently
Code:	301
Date:	Tue, 16 Jan 2024 13:27:25 GMT
Content-Type:	text/html
Connection:	close
Location:	https://jaserodley.com/robots.txt
Cache-Control:	public, max-age=31536000
CF-Cache-Status:	HIT
Age:	1197997
Report-To:	{"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=TuvQCFXqToXev0sBWVhpJABqWJnk2JGxhCBnJ0Ho7TVDHBpf4LBvcZoekOcgFFeuQIdss9bxqwgLaoD8JZEzmkef7KgHUwz43F26LpliWopS2cbISyu%2BmZ%2Flb7bAeplG4w%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL:	{"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server:	cloudflare
CF-RAY:	8466b21e99c0b91a-AMS
alt-svc:	h3=":443"; ma=86400

What steps have you taken to resolve the issue?

  1. Clearing the Cloudflare cache temporarily fixes this issue

The problem is, the issue comes back before long. The same is true for other sites on the same hosting/with the same configuration.

What are the steps to reproduce the error:

  1. Use a tool like redirect-checker.org to view https://jaserodley.com/robots.txt

Have you tried from another browser and/or incognito mode?

Yes, and using VPN connections/etc — none of this makes a difference as Cloudflare is caching the record

It’s a bit weird as it only affects the robots.txt (I guess the plugin is doing the http->https redirect for this request, but not on the rest of your site), but check your SSL/TLS is set to “Full (strict)” here…
https://dash.cloudflare.com/?to=/:account/:zone/ssl-tls

(A Cloudflare worker making the request doesn’t get the problem, so likely SSL/TLS is set to “Flexible” which forces http at the edge, even if I request https).
https://cf.sjr.org.uk/tools/check?91b5cdd6935844e9a304ad33a528d513#connection-server

2 Likes

Changing to Full (strict) worked really well for me @sjr. Thanks for making that suggestion!

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.