Download of robots.txt failed for bing


#1

Hi!

Bing can get access to my robots.txt.
Works fine for Google and other services.

I have test to create cache rules. To ignore robots.txt etc
I have create a firewall rule to accept boot traffic

url, https://www.smurfhits.se/robots.txt

Kind regards
Johan


#2

What error message does the bing bot give you? It should say what error code or issue it’s having when dealing with your robots.txt file.


#3

Only get message, ”Download of robots.txt failed for bing”

When i disable https for robots.txt bing downloads the index. But can then not download the site. Seams to be issue with the flexible SSL.

Origin is set to only response to http. So it can be the common issue with loops