How-to Download my list of blocked IPs?

Security Spectrum
#1

Hello all,

I have a similar question to this old thread Download list of IPs we blocked?

Have tried the curl commands, and used existing or newly setup API key to no avail in getting (get/download) my custom lists of ip’s setup under my account.
Which is also seen under Manage Account | Configurations | Lists.

Currently I have 1 used IP lists with 9,999 of 10,000 items/IP setup which I like to GET/Download and refined to clear out old entries…
Here error I am getting in console, and I have FREE user account.
`[email protected]:~ $ sudo curl -X GET “https://api.cloudflare.com/client/v4/accounts/01a7362d577a6c3019a474fd6f485823/rules/lists/2c0fc9fa937b11eaa1b71c4d701ab86e” \

 -H "X-Auth-Email: [email protected]" \
 -H "X-Auth-Key: 798d7s897dsa98dsa789dsa798saf789d7f89" \
 -H "Content-Type: application/json"

{“success”:false,“errors”:[{“code”:10000,“message”:“Authentication error”}]}`

#2

Is that list id and account id correct?

If you’re using Cloudflare Rule Lists then you’re using the incorrect endpoint. You’re using endpoint to list just the Ruleset names themselves Cloudflare API v4 Documentation and not the items (IPs) within a specific Ruleset name. For specifically listing the items (IPs) in a specific Ruleset name, use the example at Cloudflare API v4 Documentation

curl -X GET "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNTID/rules/lists/YOUR_LISTID/items?cursor=zzz" \
     -H "X-Auth-Email: [email protected]" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json"

resulting output

{
  "success": true,
  "errors": [],
  "messages": [],
  "result": [
    {
      "id": "2c0fc9fa937b11eaa1b71c4d701ab86e",
      "ip": "10.0.0.1",
      "comment": "Private IP address",
      "created_on": "2020-01-01T08:00:00Z",
      "modified_on": "2020-01-10T14:00:00Z"
    }
  ],
  "result_info": {
    "cursors": {
      "before": "xxx",
      "after": "yyy"
    }
  }
}

The pagination cursor. These are provided under result_info.cursors. No assumptions should be made about its content or its length.

As pagination is involved, you can’t get all 9,999 IP address items in one call. You’d have to go through each page checking the cursors.after for a value until you get all IP addresses.

You’d have to script it to loop through and get them all. I created my own script for my clients and I use to do such

i.e.

List Rulesets by name = fail2ban (what you tried in your above endpoint)

./cf-firewall-api.sh cflist-list fail2ban
{
  "id": "66c6a1514b67416fa7f959c5d52a9eb9",
  "name": "fail2ban",
  "description": "fail2ban list",
  "kind": "ip",
  "num_items": 26,
  "num_referencing_filters": 1,
  "created_on": "2021-05-05T05:07:41Z",
  "modified_on": "2022-02-25T13:48:03Z"
}

and list all items (IPs) within that Ruleset named = fail2ban

./cf-firewall-api.sh cflist-getip fail2ban
{
  "id": "33fbf98ec1a74ad080a57deb72c0fd96",
  "ip": "1.2.3.4",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:44:57Z",
  "modified_on": "2022-02-25T13:44:57Z"
}
{
  "id": "13828c5b63f94f91879a37743eafec5f",
  "ip": "1.2.3.5",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:45:11Z",
  "modified_on": "2022-02-25T13:45:11Z"
}
{
  "id": "ea3aeab953ec4ecfa87c11111693ca7f",
  "ip": "1.2.3.6",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:45:18Z",
  "modified_on": "2022-02-25T13:45:18Z"
}
{
  "id": "16acf776e76f436e9a690e1ddacdc5b3",
  "ip": "1.2.3.8",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:45:22Z",
  "modified_on": "2022-02-25T13:45:22Z"
}
{
  "id": "cf52ec241a7f4fbd8f62177f26a6ce81",
  "ip": "1.2.3.9",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:41Z",
  "modified_on": "2022-02-25T13:47:41Z"
}
{
  "id": "cd0cdeb8406146eaa486015835f5b654",
  "ip": "1.2.3.10",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:42Z",
  "modified_on": "2022-02-25T13:47:42Z"
}
{
  "id": "c36447a616ff40108d880475e1d85f60",
  "ip": "1.2.3.11",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:43Z",
  "modified_on": "2022-02-25T13:47:43Z"
}
{
  "id": "2e4fa9af27074706a314ad9d920b4691",
  "ip": "1.2.3.12",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:45Z",
  "modified_on": "2022-02-25T13:47:45Z"
}
{
  "id": "d086af3b1219464080ad716905b51216",
  "ip": "1.2.3.13",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:45Z",
  "modified_on": "2022-02-25T13:47:45Z"
}
{
  "id": "86f2b77a869a4ee1997bc1a767107c1f",
  "ip": "1.2.3.14",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:46Z",
  "modified_on": "2022-02-25T13:47:46Z"
}
{
  "id": "f6810032aea94f6ab4a364088c63798c",
  "ip": "1.2.3.15",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:47Z",
  "modified_on": "2022-02-25T13:47:47Z"
}
{
  "id": "21ae9bd798464964b911c1d53ec5d211",
  "ip": "1.2.3.16",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:48Z",
  "modified_on": "2022-02-25T13:47:48Z"
}
{
  "id": "3b4727c28948417a976a2aa009c4288b",
  "ip": "1.2.3.17",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:49Z",
  "modified_on": "2022-02-25T13:47:49Z"
}
{
  "id": "ed1056bfb541488aa6c1f258184fc808",
  "ip": "1.2.3.18",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:50Z",
  "modified_on": "2022-02-25T13:47:50Z"
}
{
  "id": "37ca3308adae4986aefdb20cdccc438d",
  "ip": "1.2.3.19",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:51Z",
  "modified_on": "2022-02-25T13:47:51Z"
}
{
  "id": "9e006e635e634219a3a4103393b9c40c",
  "ip": "1.2.3.20",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:52Z",
  "modified_on": "2022-02-25T13:47:52Z"
}
{
  "id": "5ba55d1aff464e96bde85568f6e454ac",
  "ip": "1.2.3.21",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:53Z",
  "modified_on": "2022-02-25T13:47:53Z"
}
{
  "id": "ff1131c1a1b245978cf1ea1e3a5e3c73",
  "ip": "1.2.3.22",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:54Z",
  "modified_on": "2022-02-25T13:47:54Z"
}
{
  "id": "1bd7db340bcf4e21874dbed7e22ef6da",
  "ip": "1.2.3.23",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:55Z",
  "modified_on": "2022-02-25T13:47:55Z"
}
{
  "id": "dea61680b7d14a4eb9aff9ce84d89c75",
  "ip": "1.2.3.24",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:56Z",
  "modified_on": "2022-02-25T13:47:56Z"
}
{
  "id": "ad8eeefa891447c9bb50d1061c28807e",
  "ip": "1.2.3.25",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:57Z",
  "modified_on": "2022-02-25T13:47:57Z"
}
{
  "id": "92177038eef74b368b207ccfc4e06667",
  "ip": "1.2.3.26",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:58Z",
  "modified_on": "2022-02-25T13:47:58Z"
}
{
  "id": "a1040cf4c6744643a2098dcb4607c457",
  "ip": "1.2.3.27",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:47:59Z",
  "modified_on": "2022-02-25T13:47:59Z"
}
{
  "id": "9e950c7f5ca743139b4b0b811fe327b8",
  "ip": "1.2.3.28",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:48:00Z",
  "modified_on": "2022-02-25T13:48:00Z"
}
{
  "id": "fdb40666da274555a0f6b37f833debae",
  "ip": "1.2.3.29",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:48:02Z",
  "modified_on": "2022-02-25T13:48:02Z"
}
{
  "id": "1a8f8db501814cf08a3b4bfc22f8ec1f",
  "ip": "1.2.3.30",
  "comment": "fail2ban-comment",
  "created_on": "2022-02-25T13:48:03Z",
  "modified_on": "2022-02-25T13:48:03Z"
}

Shell scripted basis is to

  1. First get the Ruleset list id using curl and jq json tool
listname=fail2ban
cfaccount_id='YOUR_ACCOUNT_ID'
cfuser='CF_EMAIL'
cftoken='CF_GLOBAL_APIKEY'
endpoint_target='https://api.cloudflare.com/client/v4/accounts'

listid=$(curl -4sX GET "${endpoint_target}/${cfaccount_id}/rules/lists" -H "X-Auth-Email: $cfuser" -H "X-Auth-Key: $cftoken" -H 'Content-Type:application/json' | jq -r --arg lname $listname '.result[] | select(.name == $lname) | .id')
  1. Use the obtained listid to get the Ruleset’s items for the 1st page and save to a temp file and inspect the cursors.after value to determine if there are more pages to list and if there is loop through all available Ruleset item pages saving them all to individual temp logs which get concatenated to list all items (IPs) at the end.
curl -4sX GET "${endpoint_target}/${cfaccount_id}/rules/lists/${listid}/items" \
     -H "X-Auth-Email: $cfuser" -H "X-Auth-Key: $cftoken" -H 'Content-Type:application/json' | jq -r > "$cftemplogdir/${listid}-page1.log"
# check for pagination past the first 25 results
after_cursor=$(cat "$cftemplogdir/${listid}-page1.log" | jq -r '.result_info.cursors.after')
page=1

while [[ "$after_cursor" != 'null' && "$after_cursor" ]]; do
  page=$(($page+1))
  curl -4sX GET "${endpoint_target}/${cfaccount_id}/rules/lists/${listid}/items?cursor=$after_cursor" \
     -H "X-Auth-Email: $cfuser" -H "X-Auth-Key: $cftoken" -H 'Content-Type:application/json' | jq -r > "$cftemplogdir/${listid}-page${page}.log"
  after_cursor=$(cat "$cftemplogdir/${listid}-page${page}.log" | jq -r '.result_info.cursors.after')
done
# concatenate all $cftemplogdir/${listid}-page* to return the results
cat $cftemplogdir/${listid}-page* | jq -r '.result[]'

For 10k items, that’s 400 pages! FYI, CF API is rate limited to max 1,200 requests per 5 min period.

2 Likes
#4

Thanks for detailed response and provide the basis of the bash script you use to help in my efforts.
I have tried the above code and first ‘curl’ command you list above and now get following output:

[email protected]:~/ $ curl -X GET "https://api.cloudflare.com/client/v4/accounts/hjsdahjsakdhjsdak579c93e85be6fsdjkljs3454/rules/lists/ipblocks/items?cursor=zzz" \
>      -H "X-Auth-Email: [email protected]" \
>      -H "X-Auth-Key: 7328782389732897321897321fnfsdjhjsdkfhsdf" \
>      -H "Content-Type: application/json"
{
  "result": null,
  "success": false,
  "errors": [
    {
      "code": 10001,
      "message": "filters.api.not_found: could not find list ipblocks"
    }
  ],
  "messages": null
}
#6

Just seems like my list-id is the issue here, but this is what I see under: Manage Account | Configurations | Lists:

lipblocks

ID: $ipblocks
Content type: IP Address

Have tried also with or without the $ symbol and also included in single speech marks ‘$ipblocks’

#7

and here is a working GET when using ZONE_ID instead, as per your link above to examples, so I know this is working for me, as it list under the zone the ipblock list I am using within that ZONE_ID (website?)

[email protected]:~/test $ curl -X GET \
> "https://api.cloudflare.com/client/v4/zones/87923hekhj12ehj213jkh123jhk213hjk123/firewall/rules" \
> -H "X-Auth-Email: [email protected]" \
> -H "X-Auth-Key: 7j38973218d7897dnhuwheuiwhreuiwhu"
{
  "result": [
    {
      "id": "hdsahasdjkhdsajkh90809sa8",
      "paused": false,
      "description": "allowed_ips",
      "action": "allow",
      "filter": {
        "id": "87490842309849230f492038",
        "expression": "(ip.src eq 2xx.xx.xx.xx and ip.src eq 31.xx.xx.xx and ip.src eq 94.xx.xx.xx)",
        "paused": false
      },
      "created_on": "2022-05-02T16:52:22Z",
      "modified_on": "2022-05-02T16:52:22Z"
    },
    {
      "id": "8f423989243894238d90d432j90824",
      "paused": false,
      "description": "ip-blocks",
      "action": "block",
      "filter": {
        "id": "43298j92438j24390fj8243908",
        "expression": "(ip.src in $ipblocks)",
        "paused": false
      },
      "created_on": "2022-04-28T19:01:29Z",
      "modified_on": "2022-04-28T19:01:29Z"
    }
  ],
  "success": true,
  "errors": [],
  "messages": [],
  "result_info": {
    "page": 1,
    "per_page": 25,
    "count": 2,
    "total_count": 2,
    "total_pages": 1
  }
}
#8

strange Cloudflare API v4 Documentation shows

GET accounts/:account_identifier/rules/lists/:list_id/items

account_identifier = account id

and I’m just copying and pasting from my script which also references account id

1 Like
#9

You’re using wrong end point https://api.cloudflare.com/client/v4/accounts/hjsdahjsakdhjsdak579c93e85be6fsdjkljs3454/rules/lists/ipblocks/items?cursor=zzz

/accounts/hjsdahjsakdhjsdak579c93e85be6fsdjkljs3454/rules/lists/ipblocks/items?cursor=zzz isn’t valid see above examples again

wrong end point again that is for firewalls not Ruleset List items

1 Like
#10

Like I said before, I did a test curl to ensure my details where working, hence using a different curl option for firewall rules in my case.
The problem lies, with what i believe is a wrong list_id which I don’t know where to get this as I have explained in y last post.
Bit baffling about list_id - it just doesn’t accept ipblocks or $ip-block etc…

#11

Finally, I got it, thanks for been patient with me :slight_smile:

1 Like
#12

You get the listid from step 1 in my example via curl query to CF API How-to Download my list of blocked IPs? - #2 by eva2000

listname=fail2ban
cfaccount_id='YOUR_ACCOUNT_ID'
cfuser='CF_EMAIL'
cftoken='CF_GLOBAL_APIKEY'
endpoint_target='https://api.cloudflare.com/client/v4/accounts'

listid=$(curl -4sX GET "${endpoint_target}/${cfaccount_id}/rules/lists" -H "X-Auth-Email: $cfuser" -H "X-Auth-Key: $cftoken" -H 'Content-Type:application/json' | jq -r --arg lname $listname '.result[] | select(.name == $lname) | .id')

listid variable derived from curl CF API query and using jq json tool

or not as a variable and without jq json tool

cfaccount_id='YOUR_ACCOUNT_ID'
cfuser='CF_EMAIL'
cftoken='CF_GLOBAL_APIKEY'
endpoint_target='https://api.cloudflare.com/client/v4/accounts'

curl -4sX GET "${endpoint_target}/${cfaccount_id}/rules/lists" \ 
       -H "X-Auth-Email: $cfuser" -H "X-Auth-Key: $cftoken" \ 
       -H 'Content-Type:application/json'

then find the listname’s listid from output

1 Like
#13

Using:

curl -X GET "https://api.cloudflare.com/client/v4/accounts/01a7362d577a6c3019a474fd6f485823/rules/lists" \
     -H "X-Auth-Email: [email protected]" \
     -H "X-Auth-Key: c2547eb745079dac9320b638f5e225cf483cc5cfdda41" \
     -H "Content-Type: application/json"

response:

[email protected]:~/test $ curl -X GET "https://api.cloudflare.com/client/v4/accounts/7asd987asd987dsa987dsa/rules/lists" \
>      -H "X-Auth-Email: [email protected]" \
>      -H "X-Auth-Key: 879sda987sda987sda987dsa978" \
>      -H "Content-Type: application/json"
{
  "result": [
    {
      "id": "c39fdcdfc5424398b7421a0c6a7d8973",
      "name": "cpanelipblocks",
      "kind": "ip",
      "num_items": 9999,
      "num_referencing_filters": 20,
      "created_on": "2022-04-24T07:54:18Z",
      "modified_on": "2022-05-20T07:14:45Z"
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}
#14

c39fdcdfc5424398b7421a0c6a7d8973 is your listid

1 Like
#15

Yep, I now got it working within a bash script using your above code too, thanks very much.
Now to get the IP’s paginated with a delay like you mentioned be fo 1200 items per 5mins.
I am making hard work of this:

/me face phalms

Pheww…

#16

Thank you @eva2000
With regards to your script, is available on git to download as I am getting stuck with jq error that comes up.
JQ is installed, but not sure why is reporting the following:

[email protected]:~# bash /home/pi/cf_blocked_ips3.sh
{
  "result": [
    {
      "id": "c39fdcdfc5424398b7421a0c6a7d8973",
      "name": "cpanelipblocks",
      "kind": "ip",
      "num_items": 9999,
      "num_referencing_filters": 20,
      "created_on": "2022-04-24T07:54:18Z",
      "modified_on": "2022-05-20T07:14:45Z"
    }
  ],
  "success": true,
  "errors": [],
  "messages": []
}
jq - commandline JSON processor [version 1.5-1-a5b5cbe]
Usage: jq [options] <jq filter> [file...]

        jq is a tool for processing JSON inputs, applying the
        given filter to its JSON text inputs and producing the
        filter's results as JSON on standard output.
        The simplest filter is ., which is the identity filter,
        copying jq's input to its output unmodified (except for
        formatting).
        For more advanced filters see the jq(1) manpage ("man jq")
        and/or https://stedolan.github.io/jq

        Some of the options include:
         -c             compact instead of pretty-printed output;
         -n             use `null` as the single input value;
         -e             set the exit status code based on the output;
         -s             read (slurp) all inputs into an array; apply filter to it;
         -r             output raw strings, not JSON texts;
         -R             read raw strings, not JSON texts;
         -C             colorize JSON;
         -M             monochrome (don't colorize JSON);
         -S             sort keys of objects on output;
         --tab  use tabs for indentation;
         --arg a v      set variable $a to value <v>;
         --argjson a v  set variable $a to JSON value <v>;
         --slurpfile a f        set variable $a to an array of JSON texts read from <f>;
        See the manpage for more options.
#17

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.