Problems with web crawlers not respecting robots.txt file

[fritex]

@josephlevin

I was getting too much web crawling traffic to the WordPress installations

I am sorry to hear that.
I also receive daily thousands of requests (user agents like bing, yandex, iar-something, admantex, dotbot, semrush, ahrefs, MJ12bot, seekport, python-something, petalbot, huawei something, blexbot, etc), but block most of the requests from them either by their user-agent or AS number.

Nevertheless, many crawlers do not respect what is written into the robots.txt file.
Furthermore, many of them just go directly to try if they can access the sitemap.xml or sitemap_index.xml file from which they just crawl-up the URLs.

@josephlevin

Is there any way to forcibly prevent the excessive crawling without doing the simple/stupid option of deleting my WordPress installations

I believe yes.
Kindly, apply the Firewall Rules as needed using below articles which contain mostly all of the “bad bots” including the crawlers, etc.:

Top 50 user agents to block

Here’s a list from the perishablepress.com 7G .htaccess firewall: (360Spider|acapbot|acoonbot|ahrefs|alexibot|asterias|attackbot|backdorbot|becomebot|binlar|blackwidow|blekkobot|blexbot|blowfish|bullseye|bunnys|butterfly|careerbot|casper|checkpriv|cheesebot|cherrypick|chinaclaw|choppy|clshttp|cmsworld|copernic|copyrightcheck|cosmos|crescent|cy_cho|datacha|demon|diavol|discobot|dittospyder|dotbot|dotnetdotcom|dumbot|emailcollector|emailsiphon|emailwolf|exabot|extract|eyenetie|feedfinder|flaming|…

Top 50 user agents to block

Good list, thanks. I have deployed that but removed python and demon (those seem to block some RSS feedreaders, YMMV). What I also have in place is this: (http.user_agent contains “SemrushBot”) or (http.user_agent contains “AhrefsBot”) or (http.user_agent contains “DotBot”) or (http.user_agent contains “WhatCMS”) or (http.user_agent contains “Rogerbot”) or (http.user_agent contains “trendictionbot”) or (http.user_agent contains “BLEXBot”) or (http.user_agent contains “linkfluence”) or (http.us…

The ASNs list to block - not all, but some of them at least, here:

Help Needed Please To Exempt Google Bots From Firewall Rules

There’s a list of ASN belongs to hosting providers:

You could also create a Firewall Rule to allow the good bots using cf.client_bot field/option.