Google Search Console 5xx errors

General
, , ,
1

Ok sorry I know I posted yesterday, just very frustrated as I haven’t a clue how to fix errors in search console and it seems googlebots cannot read my site properly.

Going to start with a basic post here and this is a few of the recent messages Google has sent me in regards to the problems.

1st message was To owner of https://www.topdogsweepstakes.com/,

Search Console has identified that your site is affected by 1 new Index coverage related issue. This means that Index coverage may be negatively affected in Google Search results. We encourage you to fix this issue.

2nd message I received yesterday was Googlebot has identified that an increased number of URLs on https://www.topdogsweepstakes.com/ return a server error HTTP 5xx. As a result, users might not be able to see your content at all, and Google might not be able to index and show your pages in search results.

I have a slew of errors in my search console
649 server errors
1 soft 404 error
15 access denied errors
48 not found errors

they show as 500, 404 and 403 errors.

I had a few really helpful responses but I am so technically challenged its ridiculous lol.

I am considering paying for a service to help fix the issue but I am very limited budgetwise and prefer to fix myself.

Any other ideas or thoughts on this anyone? OR does anyone have any suggestions on a payed service to help that is budget friendly.

Thanks kindly
Kevin

2

I share your frustration, as I just can’t figure out why Google Search Console doesn’t like your site.

A great source of support is their Webmaster Help Forum:
https://productforums.google.com/forum/?visit_id=0-636637165575166352-782034091&p=community&rd=1#!forum/webmasters

3

Hey Sdayman thank you again, will see about this forum. I am completely frustrated, seems weird to me as everything has been fine for a month and suddenly its all wacked.

I will let you know if anyone is able to help or point to a resolution.

Thanks for your support.

Kevin

1 Like
4

Try setting the indexation rate to the slowest possible. 5xx errors can happen when a single IP (Cloudflare’s) hits the firewall in front of your server for many many requests. The firewall sees it as suspicious behavior and start throwing back 5xx errors.

1 Like
5

Hi Jules thanks for your response, you are the only one who has suggested anything different the past few days, I have been everywhere seeking help.

Can you tell me where I would set the indexation rate? I am not just slightly technically challenged lol but very much so. Do I do this in Cloudflare settings somewhere?

6

That would have to be done from the Google Search Console. Here’s something I found. Instructions are at the bottom:

7

I left that alone for now sdayman, will leave this option as a last resort. Someone also pointed out that I have dns issues as this log shows intoDNS: - check DNS server and mail server health

Is this something I can contact Cloudflare for help with or something I need to do on my own?

Kevin

8

Which DNS issue is a concern? Is it because your Cloudflare name servers allow recursive queries? I’m pretty sure Cloudflare knows about that and it’s intentional. It’s not going to cause you any problems if someone uses those for their name servers.

9

Ok thanks Sdayman you would know more than I lol

Much appreciated.
Kevin

1 Like
10

Hey sdayman, I figured I would message you as have always been helpful when I am asking questions. I promise I will leave you alone soon LOL.

Just noticed when I am in Cloudflare, I click on configurations and then click on custom pages I see all these errors. Is this something to worry about, I noticed it about 2 or 3 days ago but forgot all about it in trying to diagnose all my crawl errors. I copied the page information here :

IP/Country Block
Customize the error page shown to your visitors when they visit from a blocked IP address or country.

An error has occurred, but don’t worry, we’re on it!
Help
WAF Block
Customize the error page presented to your visitors when they trigger a Web Application Firewall rule.

An error has occurred, but don’t worry, we’re on it!
Help
500 Class Errors
Customize the error page your visitors will see when a 500 class error occurs.

An error has occurred, but don’t worry, we’re on it!
Help
1000 Class Errors
Customize the error page your visitors will see when a 1000 class error occurs.

An error has occurred, but don’t worry, we’re on it!
Help
Always Online™ Error
Customize the error page your visitors will see when Always Online does not have a cached copy of the requested page.

An error has occurred, but don’t worry, we’re on it!
Help
Basic Security Challenge
Customize the Basic security challenge page presented to your visitors.

An error has occurred, but don’t worry, we’re on it!
Help
WAF Challenge
Customize the challenge page presented to your visitors when they trigger a Web Application Firewall rule.

An error has occurred, but don’t worry, we’re on it!
Help
Country Challenge
Customize the challenge page presented to visitors from countries you have selected to block.

An error has occurred, but don’t worry, we’re on it!
Help
I’m Under Attack Mode™ Challenge
Customize the challenge page presented to your visitors when you have enabled I’m Under Attack Mode.

An error has occurred, but don’t worry, we’re on it!
Help
429 errors
Customize the too many requests response presented to your visitors when they have triggered rate limiting.

An error has occurred, but don’t worry, we’re on it!

11

Those custom pages are only for paid plans. I remember see that error recently, but I just checked my free and Pro plan sites and am not getting those errors.

Is it still broken for you? Are you on a free, or a paid plan?

12

Yeah still broken, I deleted all the crawl errors yesterday after testing and fetching. Seemed to work and then this morning another 286 errors have reappeared. I had another member from another trouble shooting site tell me he had all of the exact same errors and that his was a configuration issue in Cloudflare. Suggested I contact them to go through my settings, which I have but I have not heard any responses back as of yet.

I am on a free plan.

13

Ok, then don’t worry about those Custom pages. Bummer about the crawl errors. Did that other person mention which settings cause these errors?

14

Yeah just not sure why or whats going on. Here is what his response was :

as a fellow 5XX-error-code victim, here’s what I’ve learned about this issue:

This is fundamentally a Cloudflare-related issue, though the problem may be caused at the origin servers, either by some of your hosting company firewalls, or by the firewall in security plugins such as Wordfence.

Never mind about the many 500 error page in the aw-snap.info report, because typically when a bot tries to get too many pages off Cloudflare’s CDN servers at the same time, some sort of limit is reached, and this results in the 5XX-error codes. It may be that when Cloudflare tries to get all those pages from your origin server (your hosting provider), it takes too long for it to respond, and Cloudflare is not gonna wait. Or it may be that Cloudflare itself has a firewall rule or limit, or that some plugin such as Wordfence (if you use it) has a firewall rule that limits the access of bots to your site. If you start asking around, get ready for the blame game. What you may want to do is to whitelist any online tools and legit bots you may use, such as Googlebot, Spider Frog, aw-snap.info etc) from the firewalls at both the Cloudflare level and at the origin level (Wordfence or any other security plugin, whatever firewall your hosting provider may have etc.)

Be patient, because even after you whitelist them all, there’s still a good chance the 5XX-error pages continue to appear. Try to use these bots in off-peak hours, that helped me.

If the issue persists, you may want to contact Cloudflare support, they will help you figure out what may be wrong with your configuration.

As for your DNS, I saw nothing wrong with that report. All Cloudflare-protected websites will have a similar report. The missing MX record is missing for a reason: you don’t have one, since you use a Gmail address instead of a [email protected] address.

Your robots.txt is fine, except that I don’t see any reason for allowing/disallowing wp-admin or admin-ajax.php. You should have other protections against unauthorized access to the /wp-admin/ area, either via Cloudflare, Wordfence, or some other security tool. Someone with website security expertise may be able to help you with that, if you ask in the appropriate forums or even on Twitter.

15

The mention crawling your site too quickly. Did you make any headway with slowing down the crawl rate?

16

No I really didn’t want to slow it down, was trying to sort it out before I went this route, but I may try that today.

1 Like
17

Top Dog, please see this reference:

1 Like
18

This topic was automatically closed after 14 days. New replies are no longer allowed.