Google bot requests 100% cpu usage

Website, Application, Performance Security
1

What should I do when google bot uses full cpu?

If I turn this feature on, my CPU is normal, if I turn it off, the CPU is 100% and the website is slow.
Number of visitors is 200
But Googlebot will be blocked, causing my website not to display
I found this article

def set_cache_control
return unless bot_request?
request.session_options[:skip] = true
expires_in 1.day, public: true, must_revalidate: true, stale_while_revalidate: 1.minute
end

but I don’t know where this code is inserted, can anyone help me with screenshot ?

Screenshot of the error

Screenshot 2024-11-25 140618.png
Screenshot 2024-11-25 140618.png1963×172 11.5 KB

2

helpme

3

helpme

4

Can you check in your server logs which IP address these requests are coming from? Some Cloudflare features like Automatic Signed Exchanges (SXGs) might cause heavy load on your origin, and the IP would not be a Google IP in that case.

5

image
image1398×415 27.6 KB

I’m sure it’s coming from googlebot. When I turned on challenges for the bot, the site was fine

6

That screenshot didn’t capture the IP address of the bot.

1 Like
7 
＇66.249.79.72 172.71.183.81 - - [28/Nov/2024:16:11:42 +0100] ＂GET /rapper-342-2/ HTTP/2＂ 200 17912 ＂-＂ ＂Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)＂＇
8

So is there no way to cache Google bot?

9

Sorry, I forgot to reply yesterday.

You might want to look into deploying some sort of local cache on your server. With such a low amount of traffic, Cloudflare cache isn’t really going to be of much use to you.

1 Like
10

yes I have deployed redis, but googlebot request does not hit cache