Noob question about speed tests

If pages were cached properly on a server, and I’ve cloned that server HDD to use on another server, shouldn’t the speed tests display the same results on both servers?

The thing is that I’ve had to replace my old server with a new one. On the old server, testing the speed using Google speedtest was very fast. On the second server however, the speed dropped according to Google.

The caching works fine and so the site/page should load from the Cloudflare cache, so it shouldn’t even matter the server configuration I suppose.

What can be the issue, what am I missing? The site I’m testing is the same as before, no changes.