How to Allow Bots Search Engine on Page Rules? Please hlep!

all

#1

Hi,

How do you set the Page Rules so that search engine bots like Google, Bing, Yandex and Yahoo aren’t blocked by WAF or page rules?

how do i set it up?


#2

Bots aren’t blocked by default. Did you add something that’s blocking them?


#3

Yes. I blocked IP from USA. and Googlebot is also blocked.


#4

Yes. I blocked IP from USA. and Googlebot is also blocked.


#5

With in you can implement this slightly leaner and I also took the liberty to add the bot check

(ip.geoip.country in {"CA" "CN" "GB" "IE" "RU" "TT" "US"} and not cf.client.bot)


Rate Limiting: Allowing good bots (e.g. googlebot) similar to Firewall Rules. Possible?
CSS glitch with firewall "is (not) in" rules
#6

Is the setting like in the picture, bot search engine Googlebot, Yahoo and Bing will not be blocked?


#7

Doesn’t quite match the screenshot from @sandro, it is shown there with ‘AND’ not ‘OR’ known bots - OFF


#8

What is true like this screenshot?
If I’m setting like a screenshot, is it sure that boot isn’t blocked?


#9

Should be, that rule should block all traffic from the listed countries unless it is a known bot.


#10

Is that rule the same as the one posted earlier?


#12

Thanx you @sandro and @domjh

After I blocked IP by Country. My site’s domain cannot be shared on Facebook,com.

I tried to share one of the posts on the site to Facebook,com, but it couldn’t, there was a notification like this -> Access denied | domain.com used Cloudflare to restrict access.

How do you overcome this problem?

This is Screenshot for Notif:
aef6f8aa6a299e93c13e3a77a6b43059f840fc26


#13

Facebook is using IP Addresses in the US. You blocked access from the US.


#14

Yes, I know it.
Well, how do you overcome this problem?


#15

Remove “US” from the block list in your Firewall Rule. But keep in mind that you’ll have to allow US traffic.

There’s always the chance Facebook is using a consistent User Agent you can permit, but I don’t know what that would be.


#16

I believe that to be facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php), however keep in mind by unblocking a specific user agent you’d allow everyone to access your site who claims to have that user agent.

The more secure solution would be to additionally exclude the Facebook crawler from the block. Unfortunately it appears to come from a plethora of addresses, so the easiest approach would be to exclude the entire Facebook network. Disclaimer, will also mean Facebook employees will be able to access your site however too.

In order to implement this simply change your rule into

(ip.geoip.country in {"CA" "CN" "GB" "IE" "RU" "TT" "US"} and not cf.client.bot and ip.geoip.asnum ne 32934)

#17

Is the setting like the screenshot below correct?

(ip.geoip.country in {“CA” “TT” “GB” “US” “CN” “RU” “IE” “RO” “NL”} and not cf.client.bot and ip.geoip.asnum ne 32934)