Website cannot be indexed on Google

firewall
#1

Hello to everyone,

i’m getting issues to index my website/domain on google SERP, i tried also to add it on Search console but without successfull.
I’ve added sitemap 2 weeks ago and it’s till in “couldn’t fetch” status.
So yesterday i’ve tried “URL INSPECTION” tool and i get “URL is not on Google”, so i requested an indexing and got also that message " Indexing request rejected

During live testing, indexing issues were detected with the URL" Server error (5xx)

i’ve also Cloudflare WaF active… anyone can help me to index my website?

#2

Hi @nicola.mazzoccoli61, can you share the domain in question?

#3

i’ve solved partially the problem, i’ve disabled JS challenge on Firewall rule.
For the moment i’ve not an high traffic… but i should turn it on again when my website it’s indexed?

Or maybe i’ve wrote Rule wrong?

Blockquote

(http.host eq “www.mydomain.info” or http.host eq “mydomain.info”) and not http.request.uri.path contains “/billing/modules/servers/licensing/verify.php” and not http.request.uri.path contains “/billing/modules/gateways/callback/paypalnew.php” and not http.request.uri.path contains “/billing/modules/gateways/callback/paypal.php”

#4

@cloonan do you now how to fix that Js challenge rule?

#5

For starters, your two host equality checks can be shortened to

http.host in {"www.mydomain.info" "mydomain.info"}

Second, what exactly is that rule for? Does it allow or block? Do you have a particular reason why you check for the hostnames?

#6

@sandro I really don’t know, a guy who installed cloudflare js challenge has write that, but now he doesn’t know how to help me with that rule.
Of course the problem was JS challenge becouse now I’ve turn it off and google can crawl successfully my website. But i would like to turn it on again to be more safe… what can i write on that rule?

I suppose that how he wrote before…
/billing/modules/gateways/callback/paypal.php”
/billing/modules/gateways/callback/paypalnew.php
and /billing/modules/servers/licensing/verify.php
Should be an exeption…?

#7

Without knowing what is supposed to happen and why that rule is in place it is difficult to say something.

#8

@sandro , i really don’t know… maybe i would have firewall active on all my /billing paths, can you tell me which rule i should write?

#9

I don’t think a js challenge would come through as 5xx error.

You should not be trying to get secured pages indexed in google. Do any other pages on the site get indexed?

#11

@jules when i disabled JS challenge , google, xlm-sitemap.com etc… they have succeded to crawl my webpage… so Js challenge was the problem.
today I had to reactivate it, because I received another bruteforce attack on my www.mydomain.info/billing paths… ( a spam of request that force my VPS PU to 100%)
Maybe now google isn’t able anymore to crawl my website… so i was thinking to add JS challenge only on /billing folders into my website.

my actual rule is that:
(http.host eq “www.mydomain.info” or http.host eq “mydomain.info”) and not http.request.uri.path contains “/billing/modules/servers/licensing/verify.php” and not http.request.uri.path contains “/billing/modules/gateways/callback/paypalnew.php” and not http.request.uri.path contains “/billing/modules/gateways/callback/paypal.php”

what about if i replace " “www.mydomain.info” or http.host eq “mydomain.info”" with “www.mydomain.info/billing” or http.host eq “mydomain.info/billing”

forgive me if I’m writing nonsense things… but I’m not practical at all

#12

A wildcard should work:

http.request.uri.path ~ “^/billing$”

Leave out the hosts. Give that a try and let us know.

#13

won’t work work, i’ve also tried to write "(http.host eq “www.mydomain.info/billing” or http.host eq “myiptvitalia.info/billing”) but wouldn’t work too…
maybe i should find on of theese ways:
1- don’t let JS challenge to block google crawler
2- use JS challenge only on /billing paths (but we don’t know what write on rule…)

#14

What I posted above should be the entire rule. No hosts. Is that what you tried?

#15

yes i tried both, the rule i wrote before and only this:

http.request.uri.path ~ “^/billing$”

but with that rule i get a red error , and cannot save

#16

Regex firewall rules are only available on Business Plan and up.

#17

For what it seems you want to accomplish, based on the initial post, you could just enable “I’m under attack” mode on Dashboard > Firewall > Settings.

Then you should create a few page rules for the desired paths, with a setting “Security Level: High”. You could create one per URL or it could be just one page rule for the whole /billing/ path:

*.example.com/billing/*
Security Level: High

“I’m under attack” mode bypasses Google and other search engine by default. Also, it does not generate a Firewall Events log entry for each request, something a Firewall Rule will do.

However, any attempt to JS Challenge all visitors, either via the IUA mode or firewall rules is sure to annoy your visitors. Appeasing Google to attract visitors should not be your main goal. Since this seems to be a new website, I’m not sure why it would warrant a “challenge all” approach, especially in a moment when the website is being first launched.

I’d rather create behavior-based firewall rules to block suspicious activities. For that, you should get acquainted the Firewall documentation to get a better understanding of what each action does, how to properly combine conditions etc.

2 Likes
#19

Not working yet,
what about to add: “and not cf.client.bot”

Something like that…?

(http.host eq “www.mydomain.info” or http.host eq “mydomain.info”)
and not cf.client.bot

closed #20

This topic was automatically closed after 14 days. New replies are no longer allowed.