Robot.txt and Sitemap blocked from crawling

My blog is hosted on Google’s blogspot. When I did a site audit using semrush neither could be crawl not am worried this could affect my indexability. The robot.txt is giving me a 522 error. Is there anyway to fix this I cloudflare?

Sorry for the typos

May I ask if your Blog is submitted to any of the Search engines already? :thinking:

Do you see any of the challenged or blocked events under the Security tab → Events at the Cloudflare dashboard for your zone? :thinking:

If there is none from true Google ASN and Googlebot for example, you’re good to go.

The SemRush bot, Google, Bing, Yandex, etc. are verified and could be found on the verified bot list from the link below:

Without knowing the domain, we can’t check anything. May I ask you to post and share your domain name in bracketed dot [.] notation (example[dot]com) here so we could double-check, troubleshoot and provide some more feedback information about the issue? :thinking:

Could be a temporarry issue. Did you already went throught the troubleshooting steps related to the 522 error you’re experiencing with accessing the robots.txt file from the article below? :thinking:

Furthermore, since we don’t know the content of it, it could be a syntax or a rule is blocking some bot from further crawling or indexing your pages.

checking everything that was suggested here is my domain: scottdalega.online

this what my google robots.txt.record says

How are you forwarding the apex domain scottdalega.online to www.scottdalega.online? I think that’s where the problem is.

The redirection only works with the homepage.

Note that the site is at the www subdomain.

http(s)://scottdalega.online redirects to https://www.scottdalega.online.

This works.

But http(s)://scottdalega.online/* does not redirect to https://www.scottdalega.online/*.

Any URL beyond the homepage times out with 522 Host Error.

As such https://www.scottdalega.online/robots.txt works.

But https://scottdalega.online/robots.txt does not work.

I used a page rule in cloudflare can I just add the working www. Robot.txt to my site instead?

The devil will be in the detail.

Can you provide a screenshot of the Page Rule so we can take a look?

You should be using your canonical domain (www.scottdalega.online) every time and everywhere – whether you’re giving a URL to Google, sharing it on social media, or plastering it on the noticeboard at the neighbourhood bar.

(But you should also fix the faulty redirection.)

this what it is currently ,but am about add so a records to the dns and see if that corrects anything

I removed the redirect and added these A Records

Your Page Rule was incorrect – or, at least, incomplete. That’s why only the homepage worked. The rule could have been tweaked to work correctly.

But I can see you’ve now implemented Blogger’s own guide, and the redirection is now being handled directly by Blogger.

Good luck!

thanks!!!

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.