Should I allow Google's ASN15169 through Cloudflare's firewall?

Hi,

So it should be covered by the known bots but I only think it is a few Google Bots. I need a few others but I can’t just use a bot name or user agent since those are easily spoofed.

Also, I was not seeing any of them go through in the Firewall logs before I allowed the ASN15169 through the WAF.

Some Google bots are still getting blocked then the same bot will be allowed to pass through a second later, then blocked, etc.

I would like to know if this could be a security issue? I assume not since it is coming straight from Google’s managed server.

Also, are there any disadvantages to allowing Google’s ASN through? I am seeing 16+ thousand Google bots bypassing the firewall; does this mean Argo traffic is serving the bots as well? For some reason, our Argo usage went from an average of 4GB/day to 10GB/day literally overnight but this was a month or so after I set the ASN Firewall Rule. Would that be caused by any Google Bot?

Thanks and regards,

Tug

I block 15069 (and many others), but allow cf.client.bot

Thank you for that, I now added that rule. Is there a list available to block certain ASNs? Anything you would recommend?

Did you mean 15169?

Thanks for the help.

Sorry, yes. 15169.

As far as an ASN list, there are a few around, but I look through logs for persistent bad behavior and block obvious VPS hosts when I see enough bad behavior.

1 Like

Thank you! Why do you intentionally block the Google LLC managed server 15169? Do you also block Yandex?

I have a rule allowing cf.bot, followed by three rules blocking some ASNs:

  1. ASNs used by well known cloud servers (Azure, GCP, AWS, DigitalOcean, Vultr, Oracle:

ip.geoip.asnum in {7224 15169 8074 8075 12076 16509 64236 14061 12876 20473 31898}

  1. Two rules (because of size) based on GitHub - brianhama/bad-asn-list: An open source list of ASNs known to belong to cloud, managed hosting, and colo facilities.

ip.geoip.asnum in {3223 3561 3842 4250 4323 4694 5577 6724 6870 6939 7203 7489 7506 7850 7979 8100 8455 8560 8972 9009 9370 10297 10439 10929 11588 11831 11878 12586 12876 12989 13213 13739 13926 14061 14127 14618 15003 15083 15395 15497 15510 15626 15734 16125 16262 16276 16284 16397 16509 16628 17216 18450 18779 18978 19084 19318 19437 19531 19624 19844 19871 19969 20021 20264 20454 20473 20598 20738 20773 20836 20860 21100 21159 21321 21859 22363 22552 22781 23033 23342 23352 24482 24768 24875 24940 24961 24971 25163 25369 25379 25780 25820 27257 28753 29066 29073 29182 29302 29354 29465 29550 29691 29802 29838 29854 30083 30176 30475 30633 30693 30900 30998 31034 31103 32097 32181 32244 32475 32489 32613 32780 33070 33083 33182 33302 33330 33387 33438 33480 33724 33785 33891 34305 34971 34989 35017 35366 35415 35470 35662 35908 35916 36024 36114 36290 36351 36352 36666 36873 36887 36920 36970 37018 37088 37153 37170 37209 37230 37248 37269 37280 37308 37347 37377 37472 37506 37521 37540 37643 37661 37692 37714 38001 39020 39326 39351 39392 39572 40156 40244 40676 40824 40861 41653 41665 42160 42331 42473 42695 42708 42730 42831 43146 49505 43289 43317 43350 44050 44066 45102 45187 45470 45671 45815 46261 46430 46475 46562 46664 46805 46816 46844 47328 47447 47588 49349 49367 49453 49532 49544 49981 50297 50613 50673 51159 51167 51191 51395 51430 51731 51765 51852 52048 52173 52219 53013 53340 53559 53597 53667 53755 53850 53889 54104 54203 54455 54489 54500 54540 55225 55286 55536 55933 55967 56322 56630 56934 57043 57169 57230 57858 58073 58305 59253 59349 59432 59504 59729 59764 60011 60068 60118 60404 60485 60505 60558 60567 60781 61102 61157 61317 61440 62217 62240 62282 62370 62471 62540 62567 63008 63018 63119 63128 63199 63473 63949 64245 64484 132816 133296 133480 133752 134451 136258 197155 197328 198310 199653 199883 200019 200039 201011 201525 202053 202836 203523 203629 204196 327705 327784 327813 327942 328035 394256 394330 394380 395089 395111 395978 20248 44901 200904 53057 200532 50968 135822 55293 57286 201200 24549 39458 200000 14576 54290 206898 60117 20448 201553 54825 31472 8556 29119 60476 25532 54500 49949 51698 42442 11274 57345 54817 200019 53342 33569 201983 132425 197395 42699 31698 42612 29311 54527 63213 27175 13209 29140 27223 31659 49834 49693 30152 19133 198414 45201 31981 62605 61280 53332 61147 51109 19234 40438 58797 26978 29748 35974 262990 43021 42695 39704 62899 53281 59615 55761 52335 16973 196827 32647 14992 198968 196745 62071 15169} and not http.user_agent matches "(?i)bitlybot|zapier|feedburner|feedlybot|serendeputy"

ip.geoip.asnum in {20450 30235 47205 23881 198047 14986 17920 32275 50608 199213 262170 201862 43541 24381 10200 14708 27229 48093 42465 7598 30475 55229 7349 33251 52465 52270 45152 8477 198153 52925 61412 262978 53225 41427 53101 41369 35467 59554 52674 24611 48812 40715 201449 52321 29331 201709 53221 198432 51241 19969 56799 26277 58113 28333 42120 6718 20692 17439 132717 9925 132779 42622 6188 40819 24997 38107 36408 57363 46177 62026 61107 132869 56106 32911 24931 57669 48896 45481 132509 39839 63129 53370 25048 28747 46433 55051 18570 13955 16535 22903 9823 46945 263032 36536 50986 199733 48825 35914 33552 52236 28855 198347 40728 18120 53914 12586 55720 27640 62563 202118 9290 45887 51050 20068 49485 40374 14415 46873 14384 54555 263237 20773 53918 4851 32306 133229 28216 36236 42210 51248 49815 34649 41562 33260 24220 52347 45486 33182 53055 51290 132225 133120 42776 55799 48446 263093 56732 42399 47385 40539 42244 29302 10929 47549 200147 393326 198171 57773 47583 43472 32338 9166 62082 198651 24725 29067 197902 42418 29097 196645 56110 23535 29869 62756 26484 25926 15189 20401 24679 25128 39756 32400 9412 9667 51294 23052 28099 45693 17881 17669 17918 50926 201634 22611 54641 61102 132071 10207 45577 132070 262603 29883 24558 38279 199997 50465 14120 11235 50655 17019 31240 199481 16862 47161 56784 59791 59677 202023 199990 50872 54839 58936 11230 62310 38894 47172 262287 46260 14442 133143 197648 39451 58922 27589 42400 133393 201597 28997 60800 33322 38001 199129 197372 57752 201670 14244 22152 34541 196678 43198 47625 42331 62049 35295 42311 53589 59705 36791 14160 34432 41062 59135 201630 25260 23108 40281 31590 10532 22720 27357 33070 45187 7595 26481 29713 13926 54203 62651 63128 62838 30849 14987 47577 54334 63916 50915 21217 59816 23273 59632 29452 59795 60739 15919 49313 57879 56617 62088 45179 27597 201702 32740 58667 12617 199847 25642 14567 35278 197914 41079 1442 43620 197439 198313 42705 44398 13909 34745 24958 17971 47143 59854 57682 3722 13647 205544 4670 4766 133481 14361 23470 30823 12552 3352 37963 174 15830 203953 26496 51747}

You will have to check your firewall logs because some bots might be blocked - you will see the first rule has a few exceptions (regex so change accordingly if your account is not allowed to use it) for bitly, zapier, feedburner, feedly. Also, I have a rule before those to allow things like updown.io (using a custom user-agent) and other use cases. Make sure to allow cf.bots before blocking with these rules.

Again, use with caution. Your use may be different than anyone else’s.

3 Likes

Wow! Thank you for that, I really appreciate the time you put into that. I am going to implement that but I just have a couple of questions in regards to the expression.

  1. I assume the matches is User Agent>does not contain like so: and not http.user_agent contains "(?i)bitlybot|zapier|feedburner|feedlybot|serendeputy").
  1. I honestly have no idea about regex. Is there an ASN that I need to remove if my account is not allowed to use Regex?

Again, I appreciate your help and the detailed reply!

It is quite amazing that the Known Bots expression allows the good bots through those ASNs while the other expressions block other unknown requests from the same ASN.

Regex (“Regular Expression”) is that bit with the test for the user agent. You can remove that from the rule I’ve listed above because only Business and Enterprise can use Match in the the rule expression. Instead you can add those to the Allow rule you’d have created for known bots):

(http.request.uri.path contains "bitlybot") or (http.request.uri.path contains "zapier") or (http.request.uri.path contains "zapier") or (cf.client.bot)

Ahh, okay makes sense now. Thank you for that.

One more question about this in regards to some things I have been reading about.

Why do you block those servers besides known good bots? Wouldn’t that block legitimate traffic from humans? Or are those servers mainly used by other bad or unknown bots or other malicious intent that the firewall may not block without those rules?

Thanks!

Humans usually browse from corporate proxies or from home connections. Browsing coming from colocation, virtual servers, etc is rarely “human”.

2 Likes

I constantly get new tips from The Spamhaus Project - The Top 10 Worst Botnet Countries

This helps heaps, can see what is active in countries, asn’s and much more.

Just a good site to keep an eye on, like other posts, use data carefully in firewall rules.

3 Likes

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.