Googlebot smartphone HTTP 522

We’re experiencing form a HTTP 522 errors on googlebot smartphone. It’s only giving this error on the googlebot agent. The pages works fine from any other device/user/agent and not all the URL are affected. It started with this first URL and now it seems to be expanded to this other one:

The thing here is that somehow cloudflare is blocking this agent, as if we put cloudflare in dev mode the issue is resolved.

There is no events recorded on the firewall and we’re completly lost.

Google adwords is becoming to disapprove our ads so any help is very welcome.

¡Hola @cucutoys!

Please review and follow Community Tip - Fixing Error 522: Connection timed out

1 Like

Thanks for the tip.

But in this case the 522 is only given to the google bot crawler. You can access the pages using any browser or mobile device but the google crawler still getting the 522

This could be due to the Googlebot requesting many of your site’s resources at a rate that may be triggering rate-limiting at your server. Can ensure that Cloudflare IPs are whitelisted at your origin server?

2 Likes

Thank you Blas!

I’ll double check with hosting provider.

On the other hand, if this is the case it should be affecting all site URL and not only this couple of URLs. Or in a more random pattern (once the googlebot reach the server limit) but it return 522 every time I run the google URL inspector and only with this couple of URLs

I’m missing something?

Loads fine at my end, just tested:

Maybe something is blocking Googlebot? Like Firewall Rules, too strict Security Level, other stuff enabled at Cloudflare dashboard?

Checked with host provider and there is no limit or blocking to Cloudflare. In fact we dig into server’s logs and couldn’t event find the requests from Googlebot, neither the 522 errors.

It seems that those requests are blocked at Cloudflare and never reach the server.

This is exactly what we are eager to find, what’s the setting that it’s blocking it. But we don’t have any event at Cloudflare firewall.

@cucutoys Is this 522 error occurring on JUST on the Googlebot Smartphone? Could you send me DNS responce codes (to do this go to your cloudflare dashboard then click DNS then you’ll see DNS responce codes, it will be named query by DNS responce codes or something like that) If possible send me a screenshot of that data

Cloudflare will not block search crawlers by default, however if your hosting provider set any firewall blocks for IP’s or countries belonging to these search crawlers this would cause an issue.

Would you please try and fetch a URI within Google Webmaster Tools? The URI we want to fetch is /cdn-cgi/trace. While we usually work fine with Googlebot, if it is passing through a colo that is experiencing issues, it will return an error. A bot is just like a normal visitor, in that it still has to request your site through CloudFlare. That might be the issue we are seeing here. Once you have done the fetch, could you please provide me with the output?

Thank you,

Hi William,

That exactly the issue, and furthermore it only happens with 2 URL (so far, I didn’t test the near 6k URL we have, but tested some of them and working just fine)

You can test it on google search https://search.google.com/test/mobile-friendly?id=p63e75UvOZlIMu_IyYLqfg

On the other hand I didn’t found DNS response codes or nothing similar

Hi Blas,

I did test the https://cucutoys.es/cdn-cgi/trace on Google Search Console and get the general error: " Oops! Something went wrong. Please wait a bit and try again"

Then I tested on the URL Inspection Live test and get it fetched:

Crawl Time 18 Mar 2021, 18:19:17

Crawled as Googlebot smartphone

Crawl allowed? Yes

Page fetch Successful

Indexing allowed? Yes

Still getting this error https://search.google.com/test/mobile-friendly?url=https%3A%2F%2Fcucutoys.es%2Fcdn-cgi%2Ftrace

Not sure if it gives you a hint on something.

Ok, I was able to reproduce the 522 by sending a cURL request to your site using a googlebot agent:

> GET /es/ HTTP/2
> Host: cucutoys.es
> Accept: */*
> User-Agent: AdsBot-Google (+http://www.google.com/adsbot.html)
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 522
< date: Mon, 22 Mar 2021 15:37:52 GMT

I was also able to reproduce without the googlebot user-agent:

> GET /es/ HTTP/2
> Host: cucutoys.es
> Accept: */*
> User-Agent: 
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [238 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 522
< date: Mon, 22 Mar 2021 15:47:01 GMT

So, this doesn’t necessarily seem to be due to a user-agent related issue.

We know that a 522 can occur with the following:

  1. Before a connection is established, the origin web server does not return a SYN+ACK to Cloudflare within 15 seconds of Cloudflare sending a SYN.
  2. After a connection is established, the origin web server doesn’t acknowledge (ACK) Cloudflare’s resource request within 90 seconds.

You’ll want to check the following common causes at your origin web server:

  • (Most common cause) Cloudflare IP addresses are rate limited or blocked in .htaccess, iptables, or firewalls. Confirm your hosting provider allows Cloudflare IP addresses.
  • An overloaded or offline origin web server drops incoming requests.
  • Keepalives are disabled at the origin web server.
  • The origin IP address in your Cloudflare DNS app does not match the IP address currently provisioned to your origin web server by your hosting provider.
  • Packets were dropped at your origin web server.

The issue seems intermittent, and this wouldn’t be caused by a Firewall config from your Cloudflare Dashboard. Either a hop down the path to your origin if dropping packets, or your origin is causing this error.

I’d double-check with the hosting provider to ensure that there are no additional configurations in place that would cause packets to drop or requests to be rate-limited for Cloudflare IPs.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.