Knownbots and robot.txt conflict?

My site will not load at all until I remove this rule. Once it is off, it loads fine.

I am blocking unknown bots using a simple rule.
If not in U.S.
AND
If not known bot
THEN
Block

This has worked great until recently. I started blocking some known crawlers to save on resources such as Ahrefs or SEMrush via the robot.txt file.

Your rule will block any non-US human visitors, as well as non-US unknown bots.

Is that what you want?

robots.txt does absolutely nothing to block traffic.

Yes, so my confusion is why it would break my site.

What’s the domain?

Here’s the rule. I removed any Robot.txt rule and this is still causing errors.

Why would it work until I added a robot.txt file? Why would it stay broken after the file is empty? This is the only change associated with the site breaking.

Thanks

Following up. Tried lots of things here. Removing this rule fixes the problem immediately. Why would this be the case? What should I check?

I added support already, because last time my post was closed when it wasn’t complete yet. A mod decided that my post didn’t matter and resolved it. So I didn’t want to wait the 72 hours this time since I am going to be closed regardless.

I added support already, because last time my post was closed when it wasn’t complete yet. A mod decided that my post didn’t matter and resolved it. So I didn’t want to wait the 72 hours this time since I am going to be closed regardless.

I’m not sure what help you’re asking for. You never answered which domain it was, so we can’t assist when we can’t test.

What is “My site will not load at all”? Is it you visiting your own site?

If so, in which country are you located?

1 Like

I’m in the US. No VPN.

I would prefer not to share the domain publicly.

You’re really not giving us much to go on. How about at least a screenshot of the error when you can’t reach your site?

Unable to replicate. With what I have below, I can still reach the site over cellular. With Tor (which is a T1 country code), I get a 1020 Access Denied.

It says the error is 503

Seriously…you’re not putting much effort into this.

Here’s some info for a 503 so you can troubleshoot on your own:

2 Likes

The screenshot is just a blank screen, so I didn’t think it was very helpful.

Is there anything else I could look up? Is there a way to check to see if my site is using some sort of proxy outside the US that would cause this? I double checked the IP my host gave me and it is in New Jersey.

@MoreHelp

Someone marked my ticket solved and it is not solved. I am still having issues. The community tried to help but did not continue because I did not want to give out my domain to the public.