Lots of 404 to (~1000) deleted pages - can Cloudflare (free) help?

Website, Application, Performance
1

Recently I didn’t monitored my shared wordpress site and got >30,000 new users - which created like 40,000 posts. This slowed down the whole server. Due to the emount (and no efficient way to delete a large quantity of users/posts in WP) I deleted them in the MySQL db.

So far so good, however, I still still get a massive amount of hits to deleted pages from (I presume) Spam bots.

Is there a Cloudflare (free plan) way so that they don’t hit the server?

2

Cloudflare doesn’t know which posts will return a 404, so requests have to go to your server. Unless…

Is there some common URL pattern you’re noticing they’re all hitting that doesn’t match a valid URL?

1 Like
3

Thanks. With the few page rules I have that won’t work. I will look into it further, .htaccess might be a way…

4

Firewall Rules are also an option. You get five, but each one can be quite long and complex.

5

That is good to know. But there are really too many URL, plus many foreign script URLs.

I will see if Firewall > Bot Fight Mode > ON is helping.

One hour later… That was no use. Seems not to block anything.

6

Is there a common pattern of the bot traffic e.g. coming from similar user agent, country, ASN, etc?

7

To give you some idea what I have to deal with:

34.202.159.201 - - [09/Aug/2021:11:01:55 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3803.0 Safari/537.36"
216.19.203.17 - - [09/Aug/2021:11:02:00 -0700] "GET / HTTP/1.1" 200 9430 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4442.5336 Safari/537.36 Edg/92.0.945.11"
45.77.144.75 - - [09/Aug/2021:11:02:00 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4442.5336 Safari/537.36 Edg/92.0.945.11"
45.77.144.75 - - [09/Aug/2021:11:02:00 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4013.6064 Safari/537.36"
45.77.144.75 - - [09/Aug/2021:11:02:00 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:89.0) Gecko/20100101 Firefox/89.0"
168.151.99.44 - - [09/Aug/2021:11:02:00 -0700] "GET /aviator-sunglasses-tips/ HTTP/1.1" 404 7264 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4442.5336 Safari/537.36 Edg/92.0.945.11"
216.19.203.17 - - [09/Aug/2021:11:02:02 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "https://my-domain.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:89.0) Gecko/20100101 Firefox/89.0"
185.125.193.35 - - [09/Aug/2021:11:02:02 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "https://my-domain.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:89.0) Gecko/20100101 Firefox/89.0"
2.56.19.38 - - [09/Aug/2021:11:02:02 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "https://my-domain.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:89.0) Gecko/20100101 Firefox/89.0"
73.102.9.254 - - [09/Aug/2021:11:02:02 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36"
158.46.168.17 - - [09/Aug/2021:11:02:02 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "https://my-domain.com/" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:89.0) Gecko/20100101 Firefox/89.0"
86.145.241.15 - - [09/Aug/2021:11:02:03 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:89.0) Gecko/20100101 Firefox/89.0"
174.64.18.167 - - [09/Aug/2021:11:02:03 -0700] "GET / HTTP/1.1" 200 9430 "https://www.google.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36"
174.254.48.118 - - [09/Aug/2021:11:02:03 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4442.5336 Safari/537.36 Edg/92.0.945.11"
81.136.11.76 - - [09/Aug/2021:11:02:03 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4013.6064 Safari/537.36"
174.64.18.167 - - [09/Aug/2021:11:02:04 -0700] "GET /aviator-sunglasses-tips/ HTTP/1.1" 404 7264 "https://my-domain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4013.6064 Safari/537.36"
66.249.92.12 - - [09/Aug/2021:11:02:06 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mediapartners-Google"
3.90.247.110 - - [09/Aug/2021:11:02:19 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3803.0 Safari/537.36"
207.246.122.169 - - [09/Aug/2021:11:02:24 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/603.3.8 (KHTML, like Gecko) Version/10.1.2 Safari/603.3.8"
207.246.122.169 - - [09/Aug/2021:11:02:24 -0700] "GET /aviator-sunglasses-tips/ HTTP/2" 404 7264 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4614.8842 Safari/537.36"

or

114.119.151.64 - - [10/Aug/2021:19:21:09 -0700] "GET /five-steps-to-change-the-address-of-night-in-daegu-a-lean-startup/ HTTP/1.1" 404 7343 "-" "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)"
207.46.13.111 - - [10/Aug/2021:19:21:11 -0700] "GET /page/6 HTTP/2" 404 7343 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
66.249.66.14 - - [10/Aug/2021:19:21:22 -0700] "GET /what-i-daegu-restaurant-from-judge-judy-crazy-tips-that-will-blow-your-mind/ HTTP/2" 404 7343 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
207.46.13.98 - - [10/Aug/2021:19:21:56 -0700] "GET /v9bet-introdution-4623/ HTTP/2" 404 7343 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
40.77.167.68 - - [10/Aug/2021:19:22:09 -0700] "GET /tag/sbobet-mobile/ HTTP/2" 404 7343 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.55.39.90 - - [10/Aug/2021:19:22:18 -0700] "GET /tag/%E5%A4%A7%E9%99%B8%E6%88%90%E4%BA%BA%E5%BD%B1%E7%89%87/ HTTP/2" 404 7343 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
46.167.238.36 - - [10/Aug/2021:19:22:38 -0700] "GET /confidential-information-on-%EC%A7%84%ED%95%B4%EC%B6%9C%EC%9E%A5-that-only-the-experts-know-exist/ HTTP/1.1" 404 7343 "https://my-domain.com/" "Mozilla/5.0 (Windows NT 6.1; APCPMS=^N201610260943063673863E5829051FACE43F_150^; Trident/7.0; rv:11.0) like Gecko"

I am pretty sure they use random bots from all over. Same pattern, always different IP. I see bingbot.htm often, I don’t want to block the REAL Bingbot though.

8

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.