in order to find broken links on our website, we used drlinkcheck .com service.
when scanning the website (news website, over 7,000 posts) for broken links, it created CPU overload of 100% and even caused downtime of a few minutes (until we figured out it was this).
When checking the logs, we see requests coming from their crawler. average between 20-60 every single minute.
1 wasn’t Cloudflare WAF and Security suppose to block it for flooding? otherwise, I can use such services to target other websites to create an overload and cause damage, so how come Cloudflare didn’t respond?.
2 what’s the best way to prevent such overload in the future? of course, to be careful with legitimate bots like Google.
We have the PRO plan, using also Rate Limiting (for wp-login and xmlprc).