Super Bot Fight Mode blocking curl requests from web server and Facebook bots

Here’s an interesting problem. Super Bot Fight Mode is blocking WordPress cron requests executed with curl and scheduled with crontab from my web server :upside_down_face:

Even after adding an Allow rule in the Firewall for my web server’s ASN, my curl / cron requests are still being challenged / blocked and cannot be completed. I’ve also tried using a Bypass rule with WAF Managed Rules, but I face the same problem.

I can see that the Firewall is allowing the requests with an Action Taken of Allow, but it seems like Super Bot Fight Mode is overriding the Firewall.

I also noticed that it is blocking requests from AS32934 - FACEBOOK. Those requests are most probably coming from Facebook bots and shouldn’t be blocked.

Thanks.

1 Like

The reason why this happens is due to bot management being on a different layer as the firewall, so you can’t administrate it that way unless you are part of the enterprise customers.
Your best bet is to report this and disabling the behavior of the bot fight mode or wait for Cloudflare to fix it properly.

3 Likes

exact same / but slightly different problem at this end

We are under attack from multiple bots so have turned the bot fight mode on and it does a fantastic job.
However, it blocks many of our actual services such as our forum integration and other API requests.

Whitelisting these in the firewall does absolutely nothing and there are no controls to whitelist / configure the bot fight mode so it is equally useful and completely pointless at this stage.

4 Likes

I have some good news about this.

After opening a support ticket with Cloudflare, I learned you can use IP Access Rules to bypass Super Bot Fight Mode.

So what I did is I configured some IP Access Rules to Allow my web server (both IPv4 and IPv6), and it was then able to execute curl / cron requests successfully. I also added an IP Access Rule to Allow Facebook AS32934 ASN.

So I can now use Super Bot Fight Mode without any issues.

I hope this will help other Super Bot Fight Mode users as well.

Cheers.

5 Likes

Ah I’ve always done that for my own servers as an added precaution for false positives. Good to know that it works for this too :slight_smile:

edit: hmm double checked and I had 2 curl requests challenged from server which has whitelisted both IPv4 and IPv6 server IPs in IP Access Rules that have been in place since last year.

Querying my Wordpress blog on CF Business plan with Super Bot Fight Mode via CF Firewall GraphQL API for server’s IPv6 IP when I have definite bots challenge configured

./cf-analytics-graphql.sh ip-hrs 72 2605:7380:*

{ "query":
    "query {
      viewer {
        zones(filter: {zoneTag: $zoneTag}) {
          firewallEventsAdaptiveGroups(
            limit: $limit,
            filter: $filter,
            orderBy: [datetime_ASC]
            ) {
            dimensions {
              action
              
              
              source
              datetime
              clientIP
              clientAsn
              clientCountryName
              edgeColoName
              clientRequestHTTPProtocol
              clientRequestHTTPHost
              clientRequestPath
              clientRequestQuery
              clientRequestScheme
              clientRequestHTTPMethodName
              clientRefererHost
              clientRefererPath
              clientRefererQuery
              clientRefererScheme
              edgeResponseStatus
              clientASNDescription
              userAgent
              kind
              originResponseStatus
              ruleId
              rayName
            }
          }
        }
      }
    }",
  
    "variables": {
      "zoneTag": "zoneid",
      "limit": 100,
      "filter": {
        "clientIP": "2605:7380:*",
        
        "datetime_geq": "2021-03-26T05:47:04Z",
        "datetime_leq": "2021-03-29T05:47:04Z"
      }
    }
  }

------------------------------------------------------------------
Cloudflare Firewall
------------------------------------------------------------------
since: 2021-03-26T05:47:04Z
until: 2021-03-29T05:47:04Z
------------------------------------------------------------------
2 Firewall Events for Request IP: 2605:7380:*
------------------------------------------------------------------
2605:7380:* 6365e1b88028e1c2 403 nullxnull managed_challenge 25697 UPCLOUDUSA US ORD 2021-03-27T04:30:08Z domain.com GET HTTP/1.1 /wp-cron.php ?doing_wp_cron
2605:7380:* 6365e1b8903de1c2 403 nullxnull managed_challenge 25697 UPCLOUDUSA US ORD 2021-03-27T04:30:08Z domain.com GET HTTP/1.1 /wp-cron.php ?doing_wp_cron
------------------------------------------------------------------
{
  "results": [
    {
      "action": "managed_challenge",
      "clientASNDescription": "UPCLOUDUSA",
      "clientAsn": "25697",
      "clientCountryName": "US",
      "clientIP": "2605:7380:*",
      "clientRefererHost": "",
      "clientRefererPath": "",
      "clientRefererQuery": "",
      "clientRefererScheme": "unknown",
      "clientRequestHTTPHost": "domain.com",
      "clientRequestHTTPMethodName": "GET",
      "clientRequestHTTPProtocol": "HTTP/1.1",
      "clientRequestPath": "/wp-cron.php",
      "clientRequestQuery": "?doing_wp_cron",
      "clientRequestScheme": "https",
      "datetime": "2021-03-27T04:30:08Z",
      "edgeColoName": "ORD",
      "edgeResponseStatus": 403,
      "kind": "firewall",
      "originResponseStatus": 0,
      "rayName": "6365e1b88028e1c2",
      "ruleId": "874a3e315c344b1281ad4f00046xxxxxx",
      "source": "firewallManaged",
      "userAgent": "curl/7.29.0"
    },
    {
      "action": "managed_challenge",
      "clientASNDescription": "UPCLOUDUSA",
      "clientAsn": "25697",
      "clientCountryName": "US",
      "clientIP": "2605:7380:*",
      "clientRefererHost": "",
      "clientRefererPath": "",
      "clientRefererQuery": "",
      "clientRefererScheme": "unknown",
      "clientRequestHTTPHost": "domain.com",
      "clientRequestHTTPMethodName": "GET",
      "clientRequestHTTPProtocol": "HTTP/1.1",
      "clientRequestPath": "/wp-cron.php",
      "clientRequestQuery": "?doing_wp_cron",
      "clientRequestScheme": "https",
      "datetime": "2021-03-27T04:30:08Z",
      "edgeColoName": "ORD",
      "edgeResponseStatus": 403,
      "kind": "firewall",
      "originResponseStatus": 0,
      "rayName": "6365e1b8903de1c2",
      "ruleId": "874a3e315c344b1281ad4f00046xxxxxx",
      "source": "firewallManaged",
      "userAgent": "curl/7.29.0"
    }
  ]
}

That’s weird. As soon as I enabled IP Access Rules for my web server, curl requests started to return a 200 instead of a 403 and went through as expected.

Maybe recreate those IP Access Rules? Maybe since Super Bot Fight Mode is new, those rules need to be recreated to enable them in that layer as well. It’s a long shot, but maybe that’s all it takes.

Yeah could be a temp bug as it has only occurred twice when I enabled Super Bot Fight Mode with challenges for definite automated bots. If @bsolomon can enable botscore/src in firewall events for Pro/Biz plans, I can double check if those challenged firewall events are showing botscores = 1 or 2-29 to confirm :slight_smile:

1 Like

I noticed the same thing firewall is hindering the and blocking the main operations..

I found how to fix the problem. Have a look at the solution:

IP Access Rule isn’t working for me. I have my IPv6 /64 block set to Allow on all sites, yet I still get CAPTCHA challenged:

1 Like

This didn’t work for me either. In my case, Super Bot Fight Mode is actually blocking one of our Cloudflare Worker’s outbound requests, because apparently requests made with fetch from a Cloudflare Worker route back through the firewall, and these requests are being identified as “definite bots”.

I tried adding the IPv6 IP of the Cloudflare Worker, which is for some reason always 2a06:98c0:3600::103, to the IP Access Rules, and it did nothing.

2 Likes

not meant to work - it’s by design for Pro and Biz it seems Questions About Cloudflare Super Bot Fight Mode - #5 by eva2000

I know IP Access Rule whitelisting of IPs don’t work either for me.

2 Likes

It also blocks Zapier webhook for RSS feed!
Zapier is using AWS’s dynamic IPs and no way for me to whitelist it.

1 Like

Has anyone found a solution yet? I can’t run wordpress cron and link callback from 3rd api.

Although I added Firewall Rules, it doesn’t seem to work with Super Bot Fight Mode

1 Like

You can add the ip of your server to the list of allowed ips of CF (Firewall - Tools).
This way SBF will not block your server.

2 Likes

Oh my goodness!!! Thank you SO MUCH! This took 2 days to figure out, now wp-cron is up and running AND we have Bot Fight Mode still on! I got my cake and I’m eating too. Thank you SOOOOO MUCH janvitos!!!

1 Like