Block meta searches and crawlers

Website, Application, Performance Security
1

Is it possible to block meta searches and crawlers using your firewall or other rules? If so, can you show me an article? thanks

2

You could try with the following firewall rule expression.

(http.user_agent contains "facebookexternalhit") or (http.user_agent contains "facebookcatalog")

Blocking ASN 32934 might also work, though you might even block Facebook employees.

You could also find out if they honour robots.txt and use that as well.

3

Thanks, but I meant more like google search engines and skyscanner search engines

4

What exaclty do you want to achieve?

5

I want that crawlers from google, skyscanner or tripadvisor dont crawl my page - but rather get blocked while access.

6

Well, for starters, I would use robots.txt.

And if you really want to block something you can use cf.client.bot from Fields reference · Cloudflare Ruleset Engine docs, respectively you will need to find out how all the automated systems you want to block identify themselves and then configure blocks based on that data.

But again, I’d start with robots.txt.

7

Thanks - do you have a hint to identify the crawlers of a specific engine?

8

Can be IP address, ASN, user agent. You need to check your log files for that.

9

thanks!

10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.