API Calls, hopefully easy Read/Dump issue - stuck at 100records or less

Hiya - hopefully this is something easy I’m just brain-farting on, and someone here can spot a snafu.

TL;DR I need an easy way to quickly dump /all/ of my zones into a digestible format. I’ve already realized I can’t do that in zone format and have been doing JSON conversions. However, as we continue to migrate records into CF, I’ve today noticed that any zones with >100 records aren’t pulling properly. I’ve done a few different iterations of pagination, page polling, etc.

I’m starting to wonder if using my zone pull + v4/zones + iterate&paginate is limited by something that might be a hard-coded-full-stop, or if it’s just a me issue.

Here’s what I’m using thus far (feel free to shred my approach)

Set the Cloudflare API endpoint for DNS records

DNS_RECORDS_API_URL = '[https://api.cloudflare.com/client/v4/zones/{}/dns_records]'

headers = {
‘Authorization’: f’Bearer {API_TOKEN}',
‘Content-Type’: ‘application/json’,
}

Create a list to store DNS record information

dns_records_list =

Create an argument parser

parser = argparse.ArgumentParser()
parser.add_argument(‘input_file’, help=‘Path to the input file containing a list of zone names’)
args = parser.parse_args()

Read the input file with a list of zone names

with open(args.input_file, ‘r’) as file:
zone_names = [line.strip() for line in file]

Step 1: Iterate through each specified zone and fetch DNS records

for zone_name in zone_names:
# Retrieve the zone identifier first
zones_url = '[https://api.cloudflare.com/client/v4/zones]'
params = {
‘name’: zone_name,
}

# Make a request to get the zone identifier
response = requests.get(zones_url, headers=headers, params=params)
if response.status_code == 200:
    data = response.json()
    if data['result']:
        zone_id = data['result'][0]['id']
    else:
        print(f"Zone not found for {zone_name}")
        continue
else:
    print(f"Failed to retrieve zone identifier for {zone_name}")
    continue

Step 2: Paginate through DNS records

page = 5

per_page = 50 # Adjust this value as needed

while True:
    records_url = DNS_RECORDS_API_URL.format(zone_id)
    params = {
        'page': page,

‘per_page’: per_page,

    }

    response = requests.get(records_url, headers=headers, params=params)
    if response.status_code == 200:
        data = response.json()
        if data['result']:
            dns_records_list.extend(data['result'])

if len(data[‘result’]) < per_page:

break # All records retrieved

else:

            page += 1
        else:
            break  # No more records
    else:
        print(f"Failed to retrieve DNS records for zone {zone_name}")
        break


# Added: Print a message to indicate that DNS records are being fetched
print(f"Fetching DNS records for zone identifier {zone_name}...")

response = requests.get(zones_url, headers=headers, params=params)

# Added: Print request headers for debugging
print("Request Headers:")
for key, value in response.request.headers.items():
    print(f"{key}: {value}")

data = response.json()

# Added: Print response status code and headers for debugging
print(f"Response Status Code: {response.status_code}")
print("Response Headers:")
for key, value in response.headers.items():
    print(f"{key}: {value}")


if data.get('success') and data.get('result'):
    zone_id = data['result'][0]['id']
    dns_records_url = DNS_RECORDS_API_URL.format(zone_id)

    response = requests.get(dns_records_url, headers=headers)
    data = response.json()

    if data.get('success'):
        dns_records = data.get('result')

        # Step 2: Correlate DNS records and add them to the list
        for record in dns_records:
            name = record['name']
            proxied = record['proxied']
            record_type = record['type']

            correlation = "Proxied" if proxied else "DNS Only"

            dns_records_list.append({
                'Domain': zone_name,
                'DNS Record': name,
                'Type': record_type,
                'Correlation': correlation
            })
    else:
        print(f"Failed to retrieve DNS records for {zone_name}.")
else:
    print(f"Failed to retrieve zone information for {zone_name}.")

I’m getting 200’s in header responses, but it’s looking like I can’t simply append any ?options to the v4/zones URL – any suggestions will be appreciated.

I’m AFK for the next couple of days, hence my giving up and coming here - will reply in kind/as-able.

Thanks!

welp, i mucked up the formatting - hopefully this is clear enough though

You seem to have duplicate parts of your script. Try something like

import argparse
import json
import requests

DNS_RECORDS_API_URL = "https://api.cloudflare.com/client/v4/zones/{}/dns_records"

headers = {
    "Authorization": "Bearer <API TOKEN>",
    "Content-Type": "application/json",
}
# Create a list to store DNS record information

# Create an argument parser

parser = argparse.ArgumentParser()
parser.add_argument(
    "input_file", help="Path to the input file containing a list of zone names"
)
args = parser.parse_args()
# Read the input file with a list of zone names

with open(args.input_file, "r", encoding="utf-8") as file:
    zone_names = [line.strip() for line in file]

# Step 1: Iterate through each specified zone and fetch DNS records

for zone_name in zone_names:
    # Retrieve the zone identifier first

    # Make a request to get the zone identifier
    response = requests.get(
        "https://api.cloudflare.com/client/v4/zones",
        headers=headers,
        params={"name": zone_name},
        timeout=5,
    )
    if response.status_code == 200:
        data = response.json()
        if data.get("result"):
            zone_id = data["result"][0]["id"]
        else:
            print(f"Zone not found for {zone_name}")
            continue
    else:
        print(f"Failed to retrieve zone identifier for {zone_name}")
        continue

    # Step 2: Paginate through DNS records

    page = 1

    per_page = 10
    dns_records_list = []

    while True:
        records_url = DNS_RECORDS_API_URL.format(zone_id)

        response = requests.get(
            records_url,
            headers=headers,
            params={
                "page": page,
                "per_page": per_page,
            },
            timeout=5,
        )
        if response.status_code == 200:
            data = response.json()
            if data.get("result"):
                dns_records = data["result"]
                for record in dns_records:
                    dns_records_list.append(
                        {
                            "Domain": zone_name,
                            "DNS Record": record["name"],
                            "Type": record["type"],
                            "Correlation": "Proxied"
                            if record["proxied"]
                            else "DNS Only",
                        }
                    )
                if len(dns_records) < per_page:
                    break
                page += 1
            else:
                break
        else:
            print(
                f"Failed to retrieve DNS records for {zone_name}. Error: {response.text}"
            )
            break

    with open(f"{zone_name}_records.json", "w", encoding="utf-8") as out:
        json.dump(dns_records_list, out, indent=2)

These are the permissions needed for the token

Many, many thanks - I was able to take your changes and incorporate into the larger script. Took a bit of fiddling but you got me there :slight_smile:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.