Ever since i started using cloudflare, my google ads have stopped working do to not being able to crawl my site. I get the following from the robot.txt file coming from the cloudflare server
User-agent: *
Disallow:
SITEMAP: https://healthbot.com/sitemap.xml
I do NOT have any .txt files setup in my cloudflare DNS.
What is causing this?
If Cloudflare is blocking Google’s crawler (this shouldn’t happen with default settings), then those events would show up in the Firewall Events Log.
here is the complete response from robot.txt. Looks like it is coming from cloudflare, but I don’t see it being blocked in firewall
A 304 just means the file is cached and that file hasn’t changed since it was added to the cache.
Can you get better diagnostics from Google regarding your site crawl?
Sure. I can go back to google. But they are saying this is the problem
User-agent: *
Disallow:
SITEMAP: https://healthbot.com/sitemap.xml
Which seems to be coming from Cloudflare
Just remove
User-agent: *
Disallow:
wait for 24-48 hours. You are blocking all bots. Please clear cloudflare full cache and any cache on origin server side as well.
But I do NOT have that txt record in my DNS. I can’t figure out where it is coming from.
That’s not DNS. It’s a file on your server.
It is a file in your root folder at your server hosting.
OK. I use homestead as my host. I will contact them and hopefully the can remove it.
Thanks for your help!
To me, that would mean there’s nothing Disallowed for All user agents. I wonder why it doesn’t like the file. Maybe it wants an Allow statement.
OK. I will try Google again. Not the best support there.