The robots.txt file is locked

Website, Application, Performance
1

I had 2 sites and had the same problem; Search engines have a terrible time crawling. After 2 years, I find that the robots.txt file is locked.

[Album] Imgur: The magic of the Internet
5

I see nothing in the screenshot suggesting the file is locked. The (likely) reason for the 406 Not Acceptable error is listed.

Not sure what assistance you are requesting here.

6

Ahrefs told me to contact Cloudflare, as it is the firewall that is causing problems.

7

What firewall rules do you have in place for this site?

8

I have no waf rules

9

Why do you think this is blocked by Cloudflare?

1 Like
10

The image says it’s blocking the firewall and I have Cloudflare.

I also have the Wordfence plugin, a firewall plugin. I disabled this plugin and cleared the cache with the Supercache plugin. I deactivated all plugins and changed the theme and finally cleared the cache again with Super cache. I didn’t try clearing the cache with Cloudflare in these tests.

Both Ahrefs and hosting told me to contact Cloudflare.

11

If you have not WAF rules, then (as @sandro asked) how is this a Cloudflare issue?

It also say ModSecurity firewall which is not Cloudflare.

1 Like
12

They told me the result of a program, but also Ahrefs and hosting. But if you find out the cause of why it takes years for search engines to update data, I’m happy to follow your directions.

For me I have robots.txt file and/or cache issues.

13

Your www redirects to the root domain. So any errors on www are not really relevant, there isn’t a robots.txt on www.

curl https://www.giocone.com/robots.txt
HTTP/2 301
date: Tue, 20 Jun 2023 13:59:36 GMT
content-type: text/html; charset=UTF-8
location: https://giocone.com/robots.txt
x-redirect-by: WordPress
2 Likes
14

Do you have any useful tools to see if I have problems with the site somewhere? The SEO tools say the site is fine, but there’s no way I’ve built 2 sites and neither has had traffic. Not only that, but the search console looks like the worst program in the world.

So it’s a good idea to identify where the problem is and then go talk about resolving the situation.

15

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.