The website dropped from search results!

buco0204 · September 28, 2019, 8:37am

Hi there,

We used CloudFlare for more than 5 years, and everything was just fine until two weeks ago when one of a DDoS attack was occurred, and we activated the Under Attack mod. Immediately, after that our website disappeared from search result. Than, we unable the Under Attack mode but the website still doesn’t showing up in the search results.

We have Full SSL security enabled, checked all Firewall configuration and cache options and everything looka fine. Google crowlers are not blocked, nor IPs.

Do anyone know what else we can do to fix this issue?

Thanks.

GulshanKumar · September 28, 2019, 9:24am

Starting from basic,
Please go to Search Console and do Inspect URL to debug the output.

buco0204 · September 28, 2019, 3:15pm

Already done. I am getting the Coverage message ‘Page are redirected’. When I check it in other tools getting 307 status code. But we didn’t define any redirect for that URL.

GulshanKumar · September 28, 2019, 4:26pm

307 means Internal Redirect usually happens due to HSTS, that’s normal. It would be great if you can share orignal site url. So, I can check it properly.

buco0204 · September 28, 2019, 5:03pm

Thank you Gulshan,

I know what is 307 redirect mean, but why does it happen?

We have Full SSL activated which provides secure protocol from client to CDN as well as from CDN to origin server.

Unfortunately. I can’t provide the URL due to our client policy.

sdayman · September 28, 2019, 5:12pm

When I get a 307 in my browser, it’s because my domain is on the HSTS preload list used by many browsers, so the browser knows to automatically rewrite/redirect to https if I just typed in the domain name.

floripare · September 28, 2019, 6:23pm

An HSTS redirect would be from http to https. Google treats http and https as different properties. If you are checking the right property under Google Search Console, you should be testing for https://(www.)example.com, and therefore not see this message. Are you not checking the wrong property (http:// instead of https://) by mistake??

buco0204 · September 28, 2019, 7:43pm

Thanks Flori,

Checked with https and getting this error. I am not sure is this main problem why google doesn’t index pages properly.

Please see first post. I am suspecting that is problem with CloudFlare configuration because, the website dropped when Under Attack mode was activated.

buco0204 · September 28, 2019, 7:57pm

Does it mean that Google getting redirected page as well?

sdayman · September 28, 2019, 8:52pm

They’d better be. Redirect to HTTPS is a requirement to be on the HSTS preload list.

GulshanKumar · September 30, 2019, 1:20pm

When you enable “I am under attack”, Search Engine bots cannot discover your web page.

If you do inspect URL via Search Console you will see some 5XX response. Clearly means, a Search Engine bots cannot access the web page.

As a workaround, you can have a whitelist rule.

Caution! This is a little compromise to the effectiveness of I am under attack mode. If you have sometime, please find IP ASN of Search Engine, trusted entity and do whitelisting accordingly.

Express preview - Includes Google and Bing all common user-agents.

(http.user_agent eq "APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)") or (http.user_agent eq "Mediapartners-Google") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)") or (http.user_agent eq "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)") or (http.user_agent eq "AdsBot-Google (+http://www.google.com/adsbot.html)") or (http.user_agent eq "Googlebot-Image/1.0") or (http.user_agent eq "Googlebot-News") or (http.user_agent eq "Googlebot-Video/1.0") or (http.user_agent eq "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)") or (http.user_agent eq "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36") or (http.user_agent eq "Googlebot/2.1 (+http://www.google.com/bot.html)") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)") or (http.user_agent contains "Mediapartners-Google") or (http.user_agent contains "AdsBot-Google-Mobile-Apps") or (http.user_agent eq "FeedFetcher-Google; (+http://www.google.com/feedfetcher.html)") or (http.user_agent eq "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://support.google.com/webmasters/answer/1061943)") or (http.user_agent eq "google-speakr") or (http.user_agent eq "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; DuplexWeb-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Mobile Safari/537.36") or (http.user_agent eq "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon") or (http.user_agent eq "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "msnbot/2.0b (+http://search.msn.com/msnbot.htm)") or (http.user_agent eq "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)") or (http.user_agent eq "Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)") or (http.user_agent eq "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b") or (http.user_agent eq "Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko BingPreview/1.0b") or (http.user_agent eq "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)")

buco0204 · September 30, 2019, 2:15pm

Thank you Glushan,

Now, when Under Attack mode is deactivated i am getting a page with redirect 307 and it refers to another language version of the page. And also getting canonical what is not declared by myself.
Look at the image, please!

See image please!

floripare · September 30, 2019, 9:06pm

Hi Gulshan,

Actually, Cloudflare blog on I’m Under Attack mode states that known search engines are whitelisted by default. Whatever is blocking Google crawler is probably some other rules (IP Access, User Agent Blocking, Firewall Rules, WAF etc) or some configuration at the origin server.

We’ve also designed the new checks to not block search engine crawlers, your existing whitelists, and other pre-vetted traffic. As a result, enabling I’m Under Attack Mode will not negatively impact your SEO or known legitimate visitors.

@buco0204, do you have any page rules enabled on your site? Could you post a screenshot (after obfuscating your domain/IP as needed)?

Also, since you mentioned a language-related redirect, perhaps you should make use of an online tool that checks for redirects, as it sometimes happen that redirects created by plugins are not properly disabled when we remove the plugin.

buco0204 · September 30, 2019, 9:42pm

Hi Flori,

I checked this blog post and that is why am I confused. There aren’t any page roles defined nor Whitelist.

Redirections were checked by Google Search Console and other online tools and both returning 307. I am not sure what plug in you are talking about?

floripare · September 30, 2019, 9:56pm

Never mind. I was just speculating that you may be using some of the popular language plugins for WordPress, as I know from experience that both Polylang and WPML may leave redirects they create even after the plugin removal.

But then you’d need to investigate what is causing this language redirect, certainly not something Cloudflare would do.

system · October 29, 2019, 2:35pm

This topic was automatically closed after 31 days. New replies are no longer allowed.