Best Way to Block ALL Bot Traffic?

First of all, I know that Cloudflare offers Paid plans for Advanced Bot Management, but it is not something we are looking into right now, due to its cost. I manage multiple domains and the costs will skyrocket.

So the question is simple, what is the best way to block all bot traffic with a Free plan?

I have been analyzing my logs for some days and so far I have been able to block most of the Bot traffic via ASN firewall block. However, there is always a new ASN to add, and the process never ends. And as I see here, there are thousands of Datacenter ASNs: Hosted domains by networks - IPinfo.io

So I have several options in mind to block bot traffic:

  1. Bot Fight Mode
  2. Security Level: High
  3. Threat Score > 0
  4. HTTP Version > 1.X

Or a combination of several of them.

I will test those options, for sure, but could someone help me out to start in the right direction?

The “Bot Fight Mode” alone does not work. It blocks some of the malicious hits, but it passes through other hits, such as this one, that shouldn’t have passed:

{
  "request": {
    "cf": {
      "asOrganization": "CONTABO",
      "asn": 40021,
      "city": "St Louis",
      "clientAcceptEncoding": "gzip",
      "clientTcpRtt": 7,
      "colo": "ORD",
      "continent": "NA",
      "country": "US",
      "edgeRequestKeepAliveStatus": 1,
      "httpProtocol": "HTTP/1.1",
      "latitude": "38.62870",
      "longitude": "-90.19880",
      "postalCode": "63169",
      "region": "Missouri",
      "regionCode": "MO",
      "timezone": "America/Chicago",
      "tlsCipher": "ECDHE-ECDSA-AES128-GCM-SHA256",
      "tlsClientAuth": {
        "certPresented": "0",
        "certRevoked": "0",
        "certVerified": "NONE"
      },
      "tlsVersion": "TLSv1.2"
    },
    "headers": {
      "accept_encoding": "gzip",
      "cf_connecting_ip": "144.126.136.150",
      "cf_ipcountry": "US",
      "cf_ray": "123123123123",
      "cf_visitor": "{\"scheme\":\"https\"}",
      "connection": "Keep-Alive",
      "content_length": "485",
      "content_type": "application/x-www-form-urlencoded",
      "host": "domain.com",
      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64rv:95.0) Gecko/20100101 Firefox/95.0",
      "x_forwarded_proto": "https",
      "x_real_ip": "144.126.136.150"
    },
    "ipData": {
      "city": "Iron River",
      "country": "US",
      "hostname": "whm-002.rhinoserver.net",
      "ip": "144.126.136.150",
      "loc": "46.5644,-91.4082",
      "org": "AS40021 Contabo Inc.",
      "postal": "54847",
      "region": "Wisconsin",
      "timezone": "America/Chicago"
    },
    "method": "POST",
    "url": "https://domain.com/xmlrpc.php"
  },
  "response": {
    "headers": {
      "cf_cache_status": "DYNAMIC",
      "cf_ray": "123123123123-ORD",
      "connection": "keep-alive",
      "content_type": "text/html; charset=utf-8",
      "date": "Fri, 02 Sep 2022 07:05:46 GMT",
      "server": "cloudflare",
      "strict_transport_security": "max-age=15768000;",
      "transfer_encoding": "chunked",
      "vary": "Accept-Encoding"
    },
    "origin_time": 71,
    "status_code": 403
  }
}

Testing “Threat Score > 0” & “Security Level: High”, which I found out is the same thing:

https://support.cloudflare.com/hc/en-us/articles/200170056

Testing those filters on top of the “Bot Fight Mode” filter.

Will check results in 24 hours.

Enabling “Threat Score > 0” OR “Security Level: High”, DOESN’T do anything.

Enabling HTTP Version above 1.X does filter some bots, but there are still bots coming.

Will try to brainstorm new ways to filter out bots, until someone advise a better solution.

Walk with me through this; if it was possible to block all bots on the free plan, why would anybody pay dozens of thousands on bot management?
Even if you were to pay for bot management, bots would still find a way to bypass the protection. It’s an arms race where sometimes the bad guys win and sometimes the good guys win.

Those are valid options; however, the only decent protection listed is the Bot Fight Mode; the rest are trivial protections that aim to stop dummy crawlers/scrapers/DDoS attacks.

Bot protection is a highly challenging problem to deal with it; companies pay up to thousands of dollars per hour to have proper bot protections in place. And even then, there is always the intelligent bot that might be able to bypass the protection (or not? Depends on who is having a lucky day).

Unfortunately, small companies do not have proper solutions on the market to mitigate bot attacks. Cloudflare does a good job but can only do so much with the current technology limitations and the price each customer pays.

I have tried most solutions available in the range of $0 to $600 per month, and I can safely say that CF offers the best performance for the buck. Most products you will find out there will be a significant disappointment once you try them. It’s hard to detect snake oil on security products.

If you enter the enterprise market (>$85k per year ), there are absolutely amazing products that do an excellent job at stopping most bots; however, companies that need this level of protection are hard to find.

If you want to try some alternative products, check Stackpath, pricing starts at around $60 per month and has built-in bot protection.

I’m not promoting them; however, I’m giving you the closest competitor CF has right now for small-medium-sized companies.

Even then, you will (most likely) find yourself coming back to Cloudflare a few months later because the current pricing and services CF offers are unmatchable on the market.

You need to ask yourself why you are picking up this fight. Is it causing a loss in revenue? Does it obscure the logs you monitor? Is it due to a DDoS Attack? Are bots draining bandwidth?
If the only concern is the noise those bots cause, then I’m dubious that it will be worth your time, effort, or investment in any solution.

If you still want to move forward and need something cheap; try using nginx-test-cookie on top of Cloudflare.
You will want to add some checks/traps to the plugin and obfuscate the delivered JS; should make the overall protection slightly better. Especially if you target chromium/selenium/headless bots.

4 Likes

Hey @jnperamo thanks for your reply!

Bot traffic is causing loss in revenue because we buy traffic from multiple advertising platforms and some traffic is bot traffic, that these platforms don’t block or don’t want to block.

I was checking the “testcookie-nginx-module” (testcookie-nginx-module), but I see it uses a redirect, to load a cookie on the client’s browser. That’s definitely something not ideal, because real users will see that redirect and there are a lot of advertising platforms that don’t allow redirects.

I see Cloudflare offers 4 tiers in terms of Bot mitigation:

  • FREE: Simple bots
  • PRO: More advanced bots
  • BUSINESS: Sophisticated bots and basic bot analytics
  • ENTERPRISE: All bots, anomaly detection, custom CAPTCHAs & threat response, advanced bot analytics, and more

I see 2 types of Bot Mitigation, “Bot Fight Mode” and “Super Bot Fight Mode”. Can these 2 be combined together or is it 1 or the other?

Out of curiosity, how much does the ENTERPRISE tier cost?

nginx test cookie has multiple ways of functioning.

  1. Cookie support / Redirect.
  2. JS Challenge
  3. CAPTCHA Challenge

I suspect you have looked at the first mode, JS and CAPTCHA are the stronger contestants.

Yeah, essentially, the more you pay to CF, the more visibility and granularity they offer for the bot protection.

ENT is built up upon your needs, you can expect an entry cost of $3k per month, however, if you make it clear that you only need bot protection and don’t mind skipping the rest of addons (support, sla, etc) it miiiiiight be cheaper.

Ah… understood so it’s more of a clickfarm/fake advertising concern? So somebody has ads on their site and once they reach your site, you realize those are fake/bot clicks?

I believe there are click protection services that can interop with Cloudflare and are likely cheaper than buying a full fledged bot protection; I haven’t tested them myself but I reckon most platforms offer a free trial.
The idea behind those services is that they detect the malicious/fraudulent clicks and then you can dispute the “lost” revenue with the ads vendor.

2 Likes

Ok, I have upgraded one of my sites to PRO plan in Cloudflare to check how the Bot/Super Bot Fight Mode improves the bot detection and prevention.

However, when I access Security > Bots, I see the following screen:

Is this what I’m supposed to see? It seems like there is problem loading the info there.

Once it collects some data; you will begin seeing something like this:

3 Likes

In case you are curious; this is the business super bot fight mode:

The main drawback is that you can’t fine-tweak it.

2 Likes

So I have been testing the “Super Bot Fight Mode” for PRO users for some days. While most of the bot traffic is gone, there are still visits that are clearly bots, that are not getting filtered correctly. Some examples of these:

{
  "request": {
    "cf": {
      "asOrganization": "Microsoft Azure",
      "asn": 8075,
      "city": "Paris",
      "clientAcceptEncoding": "gzip, deflate",
      "clientTcpRtt": 1,
      "colo": "CDG",
      "continent": "EU",
      "country": "FR",
      "edgeRequestKeepAliveStatus": 1,
      "httpProtocol": "HTTP/1.1",
      "isEUCountry": "1",
      "latitude": "48.83230",
      "longitude": "2.40750",
      "postalCode": "75001",
      "region": "Île-de-France",
      "regionCode": "IDF",
      "timezone": "Europe/Paris",
      "tlsClientAuth": {
        "certPresented": "0",
        "certRevoked": "0",
        "certVerified": "NONE"
      }
    },
    "headers": {
      "accept": "*/*",
      "accept_encoding": "gzip",
      "cf_connecting_ip": "20.199.10.94",
      "cf_ipcountry": "FR",
      "cf_ray": "123123123",
      "cf_visitor": "{\"scheme\":\"http\"}",
      "connection": "Keep-Alive",
      "host": "domain.com",
      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0",
      "x_forwarded_proto": "http",
      "x_real_ip": "20.199.10.94"
    },
    "ipData": {
      "city": "Paris",
      "country": "FR",
      "ip": "20.199.10.94",
      "loc": "48.8534,2.3488",
      "org": "AS8075 Microsoft Corporation",
      "postal": "75000",
      "region": "Île-de-France",
      "timezone": "Europe/Paris"
    },
    "method": "GET",
    "url": "http://domain.com/file.php"
  },
  "response": {
    "headers": {
      "cf_cache_status": "DYNAMIC",
      "cf_ray": "123123123-CDG",
      "connection": "keep-alive",
      "content_type": "text/html; charset=utf-8",
      "date": "Mon, 05 Sep 2022 05:35:08 GMT",
      "server": "cloudflare",
      "transfer_encoding": "chunked",
      "vary": "Accept-Encoding"
    },
    "origin_time": 99,
    "status_code": 404
  }
}
{
  "request": {
    "cf": {
      "asOrganization": "Hetzner Online GmbH",
      "asn": 24940,
      "city": "Helsinki",
      "clientAcceptEncoding": "gzip, deflate",
      "clientTcpRtt": 0,
      "colo": "DME",
      "continent": "EU",
      "country": "FI",
      "edgeRequestKeepAliveStatus": 1,
      "httpProtocol": "HTTP/1.1",
      "isEUCountry": "1",
      "latitude": "60.17920",
      "longitude": "24.93370",
      "postalCode": "00131",
      "region": "Uusimaa",
      "regionCode": "18",
      "timezone": "Europe/Helsinki",
      "tlsClientAuth": {
        "certPresented": "0",
        "certRevoked": "0",
        "certVerified": "NONE"
      }
    },
    "headers": {
      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "accept_encoding": "gzip",
      "accept_language": "en-US,en;q=0.9,fr;q=0.8",
      "cache_control": "max-age=0",
      "cf_connecting_ip": "65.108.58.55",
      "cf_ipcountry": "FI",
      "cf_ray": "123123123",
      "cf_visitor": "{\"scheme\":\"http\"}",
      "connection": "Keep-Alive",
      "host": "www.domain.com",
      "referer": "anonymousfox.co",
      "upgrade_insecure_requests": "1",
      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36",
      "x_forwarded_proto": "http",
      "x_real_ip": "65.108.58.55"
    },
    "ipData": {
      "city": "Helsinki",
      "country": "FI",
      "hostname": "static.55.58.108.65.clients.your-server.de",
      "ip": "65.108.58.55",
      "loc": "60.1695,24.9354",
      "org": "AS24940 Hetzner Online GmbH",
      "postal": "00100",
      "region": "Uusimaa",
      "timezone": "Europe/Helsinki"
    },
    "method": "GET",
    "url": "http://domain.com/admin.php"
  },
  "response": {
    "headers": {
      "cf_cache_status": "DYNAMIC",
      "cf_ray": "123123123-DME",
      "connection": "keep-alive",
      "content_type": "text/html; charset=utf-8",
      "date": "Mon, 05 Sep 2022 02:48:59 GMT",
      "server": "cloudflare",
      "transfer_encoding": "chunked",
      "vary": "Accept-Encoding"
    },
    "origin_time": 295,
    "status_code": 404
  }
}

Why the “Super Bot Fight Mode” is not correctly filtering these ASNs which are clearly Datacenter ASNs?

I am still getting Bot traffic with the “Super Bot Fight Mode” enabled. Not sure if the algorithm is working correctly. It should be learning from false negatives. Here are 2 more examples of bot visits that didn’t get filtered:

{
  "request": {
    "cf": {
      "asOrganization": "Microsoft Azure",
      "asn": 8075,
      "city": "Melbourne",
      "clientAcceptEncoding": "gzip, deflate",
      "clientTcpRtt": 1,
      "colo": "MEL",
      "continent": "OC",
      "country": "AU",
      "edgeRequestKeepAliveStatus": 1,
      "httpProtocol": "HTTP/1.1",
      "latitude": "-37.81590",
      "longitude": "144.96690",
      "postalCode": "3001",
      "region": "Victoria",
      "regionCode": "VIC",
      "timezone": "Australia/Melbourne",
      "tlsClientAuth": {
        "certPresented": "0",
        "certRevoked": "0",
        "certVerified": "NONE"
      }
    },
    "headers": {
      "accept": "*/*",
      "accept_encoding": "gzip",
      "cf_connecting_ip": "20.70.88.169",
      "cf_ipcountry": "AU",
      "cf_ray": "123123123",
      "cf_visitor": "{\"scheme\":\"http\"}",
      "connection": "Keep-Alive",
      "host": "www.domain.com",
      "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0",
      "x_forwarded_proto": "http",
      "x_real_ip": "20.70.88.169"
    },
    "ipData": {
      "city": "Melbourne",
      "country": "AU",
      "ip": "20.70.88.169",
      "loc": "-37.8140,144.9633",
      "org": "AS8075 Microsoft Corporation",
      "postal": "3000",
      "region": "Victoria",
      "timezone": "Australia/Melbourne"
    },
    "method": "GET",
    "url": "http://www.domain.com/.aws/credentials"
  },
  "response": {
    "headers": {
      "cf_cache_status": "DYNAMIC",
      "cf_ray": "123123123-MEL",
      "connection": "keep-alive",
      "content_type": "text/html; charset=utf-8",
      "date": "Tue, 06 Sep 2022 04:33:50 GMT",
      "referrer_policy": "same-origin",
      "server": "cloudflare",
      "transfer_encoding": "chunked",
      "vary": "Accept-Encoding",
      "x_content_type_options": "nosniff",
      "x_frame_options": "SAMEORIGIN"
    },
    "origin_time": 252,
    "status_code": 403
  }
}
{
  "request": {
    "cf": {
      "asOrganization": "Hetzner Online GmbH",
      "asn": 24940,
      "city": "Stuttgart",
      "clientAcceptEncoding": "gzip, deflate",
      "clientTcpRtt": 28,
      "colo": "VIE",
      "continent": "EU",
      "country": "DE",
      "edgeRequestKeepAliveStatus": 1,
      "httpProtocol": "HTTP/1.1",
      "isEUCountry": "1",
      "latitude": "48.76700",
      "longitude": "9.18270",
      "postalCode": "70597",
      "region": "Baden-Wurttemberg",
      "regionCode": "BW",
      "timezone": "Europe/Berlin",
      "tlsCipher": "ECDHE-ECDSA-AES128-GCM-SHA256",
      "tlsClientAuth": {
        "certPresented": "0",
        "certRevoked": "0",
        "certVerified": "NONE"
      },
      "tlsVersion": "TLSv1.2"
    },
    "headers": {
      "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
      "accept_encoding": "gzip",
      "accept_language": "en-US,en;q=0.9,fr;q=0.8",
      "cache_control": "max-age=0",
      "cf_connecting_ip": "138.201.50.113",
      "cf_ipcountry": "DE",
      "cf_ray": "123123123123",
      "cf_visitor": "{\"scheme\":\"https\"}",
      "connection": "Keep-Alive",
      "host": "domain.com",
      "referer": "www.bing.com",
      "upgrade_insecure_requests": "1",
      "user_agent": "wp_is_mobile",
      "x_forwarded_proto": "https",
      "x_real_ip": "138.201.50.113"
    },
    "ipData": {
      "city": "Falkenstein",
      "country": "DE",
      "hostname": "static.113.50.201.138.clients.your-server.de",
      "ip": "138.201.50.113",
      "loc": "50.4779,12.3713",
      "org": "AS24940 Hetzner Online GmbH",
      "postal": "08223",
      "region": "Saxony",
      "timezone": "Europe/Berlin"
    },
    "method": "GET",
    "url": "https://domain.com//wp-signin.php?dizo&source=http://gov.co.ve/wepqgvuasyk.html&dest=wp-signin"
  },
  "response": {
    "headers": {
      "cf_cache_status": "DYNAMIC",
      "cf_ray": "123123123123-VIE",
      "connection": "keep-alive",
      "content_type": "text/html; charset=utf-8",
      "date": "Mon, 05 Sep 2022 22:23:34 GMT",
      "referrer_policy": "same-origin",
      "server": "cloudflare",
      "strict_transport_security": "max-age=15768000;",
      "transfer_encoding": "chunked",
      "vary": "Accept-Encoding",
      "x_content_type_options": "nosniff",
      "x_frame_options": "SAMEORIGIN"
    },
    "origin_time": 116,
    "status_code": 404
  }
}

Is this issue related? Cloudflare Status - Cloudflare Bot Management issue

Pro plans do not have the business feature of machine learning, if you want to have full protection, business is what you need to look at next, for 200 bucks plus all the other extras that’s cheap compared to data dome and others. I may be wrong.

But even with business, you cannot modify it, that’s where you need to upgrade again to enterprise.

This may help machine learning page, business/enterprise, etc.

To be honest, I was expecting a better Bot protection with the PRO plan. There’s no way I am paying 200€/mo for Bot protection. Come on…Microsoft and Hetzner are Datacenter ASNs. These are clearly bots, it’s difficult to believe that Cloudflare can’t block these with “Super Bot Fight Mode”.

Pro is just basic, its just a starting for you to upgrade, like all of CF services. Send a ticket in about it, you will be prompted not to upgrade to business, but to enterprise :wink: lol.

First off, I wouldn’t advise blocking ALL bots (as it’s a waste of time - as stated by @jnperamo )!

Although I would advise trying (not cf.client.bot) or (cf.client.bot) (but there are no guarrentes this will work)! As mentioned:

Yeeeah, unfortunately that part is wacky. I understand your point and I definitely wouldn’t pay $20/month if the bot protection was my only interest.
Neither I would pay $200/month on its current state; the only option that is somewhat worth the cost is the enterprise version and we all know only few companies can afford that.

Blocking ASNs is appealing since a chunk of the traffic is annoying; however, it’s not as easy due to the following:

  1. Many legitimate bots use those ASNs to hide their IPs to avoid people spoofing the content to crawlers/bots.
  2. Many companies have enterprise VPNs that expose those IPs and only those IPs. Those VPNs aren’t meant for privacy or anything like that but to secure and monitor employees.
  3. Many companies are adapting remote work environments (they have dedicated services for this). and the ASN they expose are public IPs from AWS, Azure, etc.

Blocking ASNs definitely kills many malicious/annoying bots but it also carries its own consequences;

CF bot protection tries to be “permissive” so; between blocking a legitimate human and blocking few more bots vs allowing more bots and being less intrussive; CF picks the option that makes browsing easier for humans.

There is this project: https://www.crowdsec.net/ which seems to try to fight the issues you are facing; it’s free by default and might be able to help with the issues you are facing.
The main hesitation is that AFAIK it requires an agent to run on your machine, which carries its own issues if you are concerned about running third party code on your system (its open source though).

2 Likes

I’ll need to implement my own solution to block bots.

It’s a pity because Cloudflare Super Bot Fight Mode seems good.

But when you test it, it’s not good enough.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.