The robots.txt file is locked

I had 2 sites and had the same problem; Search engines have a terrible time crawling. After 2 years, I find that the robots.txt file is locked.

I see nothing in the screenshot suggesting the file is locked. The (likely) reason for the 406 Not Acceptable error is listed.

Not sure what assistance you are requesting here.

Ahrefs told me to contact Cloudflare, as it is the firewall that is causing problems.

What firewall rules do you have in place for this site?

I have no waf rules

Why do you think this is blocked by Cloudflare?

1 Like

The image says it’s blocking the firewall and I have Cloudflare.

I also have the Wordfence plugin, a firewall plugin. I disabled this plugin and cleared the cache with the Supercache plugin. I deactivated all plugins and changed the theme and finally cleared the cache again with Super cache. I didn’t try clearing the cache with Cloudflare in these tests.

Both Ahrefs and hosting told me to contact Cloudflare.

If you have not WAF rules, then (as @sandro asked) how is this a Cloudflare issue?

It also say ModSecurity firewall which is not Cloudflare.

1 Like

They told me the result of a program, but also Ahrefs and hosting. But if you find out the cause of why it takes years for search engines to update data, I’m happy to follow your directions.

For me I have robots.txt file and/or cache issues.

Your www redirects to the root domain. So any errors on www are not really relevant, there isn’t a robots.txt on www.

HTTP/2 301
date: Tue, 20 Jun 2023 13:59:36 GMT
content-type: text/html; charset=UTF-8
x-redirect-by: WordPress

Do you have any useful tools to see if I have problems with the site somewhere? The SEO tools say the site is fine, but there’s no way I’ve built 2 sites and neither has had traffic. Not only that, but the search console looks like the worst program in the world.

So it’s a good idea to identify where the problem is and then go talk about resolving the situation.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.