What custom rules on cloudflare are you using to stop bots / scrappers

Website, Application, Performance
1

This is a general question.
What custom rules on cloudflare are you using to stop bots / scrappers / A.I. to crawl your content?

I am really under pressure to stop all those bad bots and crawlers from my page, because they are not only stealing my content, they attack my website with bad reputation links.
Beside activating the

  • BOT FIGHT MODE
  • BLOCK AI and scrappers

(which might not block everyhting out)
I have done the following
So I have set up those custom rules:

  1. verified AI Crawler = blocked
    (cf.verified_bot_category eq “AI Crawler”)
    BLOCK

  2. If it is a verified BOT (google for example), I SKIP rules 3
    (http.user_agent contains “Google-InspectionTool”) or (cf.verified_bot_category eq “Search Engine Crawler”) or (cf.verified_bot_category eq “Monitoring & Analytics”) or (cf.verified_bot_category eq “Page Preview”) or (cf.verified_bot_category eq “Aggregator”) or (cf.verified_bot_category eq “Search Engine Optimization”)
    SKIP

  3. I challenge all countries that not in my portfolio - market. My market is only German speaking countries, so the rule is:
    (not ip.geoip.country in {“AT” “DE” “LI” “LU” “CH”})
    MANAGED CHALLENGE
    (0.54% manage this challenge…shows you that this is realy helpful…or? everyone outside my geman countries is a bot)

Will this help to block all stupid crawlers from the USA, INDIA, RUSSIA, CHINA? I hope so.

Any better ideas?

show me your best rules to have stupid bots and crawlers and scrappers blocked.

2

This one might help, if can be use you might need to check and tune it based on what you want.

3

This topic was automatically closed after 15 days. New replies are no longer allowed.