Google can't crawl the website

[serge]

Hi!

I’m trying to crawl our website in Google Search console, but it says that crawling is blocked by robots.txt while It’s not.
This our website: https://www.outstandly.com/

I’ve reached to google and they suggested to get in touch with our hosting and Cloudflare to see if it’s not the issue on your side.
I’ve already got a reply from Cloudways that from their side with the server, htaccess and robots.txt files everything is good.

So I’m curious maybe you can help me with this matter?
Thank you so much!

[sandro]

Does your firewall event log show any blocked requests for robots.txt?

Also, post a screenshot of the exact message you receive in the console.

[serge]

Hi Sandro,

Thank you for your time looking into it!
Nope, firewall event log says ‘‘No Firewall events’’

Here is a screenshot:

And this is my message on Google webmasters forum: "Crawl blocked by robots.txt" while it's not? - Google Search Central Community

[sandro]

The person you were talking to was not a Google employee though, but just a regular forum member. If possible I’d still reach out to Google and clarify it with them. The message seems to indicate they can reach the file but it blocks crawling which does not seem to be the case.

Additionally you could try to explicitly allow the other paths in your robots.txt.

[serge]

Thanks!
I’ve tried contacting Google support in chat, but the person there was extremely not helpful at all…

I’ve tried also your suggestion to explicitly allow the other paths (thanks for the idea), but unfortunately, it didn’t worked out…

[Xaq]

You are using a dummy sitemap generated by Yoast SEO. Maybe error message indicates non of the URLs in sitemap can be resolved.