We have been running Cloudflare (free plan) on two websites that are both located on the same server for many months now. All seems to be working fine. Over the last six months or so we have seen worse and worse search results for one of the websites, while the other has remained without problems. We have suspected and investigated many different things such as possible hacks, not being mobile friendly etc. Please also note that I am a layman in this area and I apologize ahead of time for any stupidies here.
Google Search Console is reporting thousands of soft 404 and pages with errors and even blocked by robot.txt, although we have triple checked and nothing is blocked.
I am aware that there can be many different reasons for Soft 404. Over the last few days, we’ve been looking into if we may have misconfigured Cloudflare and one item in particular that in my layman’s eyes could be the culprit of the soft 404 is that when using Google Fetch and Render function to see what Google sees, the site with the problem is not displaying the whole webpage, only the Cloudflare part, although the rendered image looks just fine.
Here is a Fetch from both sites. The is identical or close to identical (haven’t checked carefully) for every page we have run fetch and render, thus if that’s what Google sees, that should be a good explanation of why they don’t accept our pages, but rule them out as Soft 404.
Working site at top:
So a couple of questions
Has anyone seen this behavior of Google Fetch only displaying parts of the page (It appears fine in render and the website is fully visible and browsable, so it loads fine, the problem is only with google)
Is there a setting in Cloudflare that we have missed or set wrong for the entire code of the end-page not to pass through to google
In a side-by-side comparison, I do see a difference in the Fetched HTTP response,
the problematic website has “Cache-Control” and “Expires” which isn’t appearing on the working. Is that a setting we have missed or set incorrectly?
I did post a ticket with Cloudflare and have as of yet (24 hrs) not received a response, but I did get an automated email stating that it could be a DNS issue and that we must point it to Authoritative nameservers. We have pointed it the same way we do every website, and so I am not sure if that means authoritative, or if how it was set up with our pointing nameserver that could cause this.
Also, when comparing the settings between the two websites in Cloudflare admin area, I cannot see any differences, but I may have missed something somewhere.
Any insights or tips would be greatly appreciated.