Cloudflare blocking google crawler

Ever since i started using cloudflare, my google ads have stopped working do to not being able to crawl my site. I get the following from the robot.txt file coming from the cloudflare server

User-agent: *
Disallow:
SITEMAP: https://healthbot.com/sitemap.xml

I do NOT have any .txt files setup in my cloudflare DNS.

What is causing this?

If Cloudflare is blocking Google’s crawler (this shouldn’t happen with default settings), then those events would show up in the Firewall Events Log.

1 Like

here is the complete response from robot.txt. Looks like it is coming from cloudflare, but I don’t see it being blocked in firewall
snip1

A 304 just means the file is cached and that file hasn’t changed since it was added to the cache.

Can you get better diagnostics from Google regarding your site crawl?

2 Likes

Sure. I can go back to google. But they are saying this is the problem

User-agent: *
Disallow:
SITEMAP: https://healthbot.com/sitemap.xml

Which seems to be coming from Cloudflare

Just remove

User-agent: *
Disallow:

wait for 24-48 hours. You are blocking all bots. Please clear cloudflare full cache and any cache on origin server side as well.

But I do NOT have that txt record in my DNS. I can’t figure out where it is coming from.

That’s not DNS. It’s a file on your server.

It is a file in your root folder at your server hosting.

OK. I use homestead as my host. I will contact them and hopefully the can remove it.

Thanks for your help!

1 Like

To me, that would mean there’s nothing Disallowed for All user agents. I wonder why it doesn’t like the file. Maybe it wants an Allow statement.

1 Like

OK. I will try Google again. Not the best support there.