Daily Surge in NXDOMAIN Requests from 9AM to 10:30AM

[jack.t]

Hello,

I’ve been encountering a strange issue within the DNS logs.

Every morning, without fail, between 9:00 AM and 10:30 AM, my domain experiences an unusually high spike in NXDOMAIN requests. This has become a daily occurrence, and I’m struggling to find it’s origin.

During these times the website gets slower and occasionally presents 522 errors.

I’ve spoken to my server providers and they said they can’t do much to help as we are using Cloudflare and the DNS query logs are only available within the highest CF pricing package which I am not wanting to purchase.

Does anyone have any ideas?

Many thanks.

[sjr]

@jack.t

my domain experiences an unusually high spike in NXDOMAIN requests

Are these seen in the DNS analytics? Or reported from your browser when you try to reach your site.

If in the analytics they will be requests for DNS records that you haven’t set. How many such requests are there? If a significant number (can you show a screenshot?), this, combined with your origin server timing out (522 error) could indicate some form of DoS attack that’s overloading your origin (although strange to always be at the same time). Do you see a spike in traffic during those times? The NXDOMAIN errors won’t directly affect your origin since, by definition, they are requests that can’t reach it.

What is the domain?

You can try to find out what those DNS requests are by setting a wildcard DNS record so they resolve, but you’ll need to point that somewhere other than your origin if that’s going down during that time as you don’t want to send it any more requests. If you set it to be proxied perhaps use a Cloudflare worker to log them and write to R2 or something, or use another server, or use the WAF to block any subdomains you don’t use so such requests appear in the security log.

[jack.t]

Hello - thank you for the advice!

I have done as you suggested and set up a worker and linked to a Worker KV namespace which is now giving me various information about these requests.

This is the script I am using

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

/**
 * Respond to the request
 * @param {Request} request
 */
async function handleRequest(request) {
  const url = new URL(request.url)
  const path = url.pathname
  const ipAddress = request.headers.get('CF-Connecting-IP') || 'unknown'
  const userAgent = request.headers.get('user-agent') || 'unknown'

  // Collecting all headers (consider filtering for privacy/security)
  let headers = {}
  for (let [key, value] of request.headers) {
    headers[key] = value
  }

  // Log the request
  const timestamp = new Date().toISOString()
  const logKey = `${url.hostname}${path}-${timestamp}`
  const logValue = JSON.stringify({
    IP: ipAddress,
    Timestamp: timestamp,
    UserAgent: userAgent,
    Headers: headers, // Consider privacy/security implications
    Method: request.method,
    QueryString: url.searchParams.toString()
  })
  
  // Use the KV namespace to store the log
  await REQUEST_LOGS.put(logKey, logValue)

  // Redirect or handle the request in other ways
  //return Response.redirect(`https://mywebsite.com/`, 301)
}

Can you think of any additional data worth logging?

So far I haven’t been able to determine what the exact source of these requests are.

Here is an example of the spiking of requests in the morning:

Appreciate your help!

[sjr]

Nicely done. It will be interesting to see what you find.

(My assumption was of course that these DNS queries will result in HTTP requests, rather than some other protocol which won’t hit the worker - but at least you’ll know).

[jack.t]

I’ve been keeping a close eye on everything and I am quite confused.

Now that I have the wildcard CNAME record in place essentially all our NXDOMAIN requests are gone (as expected).

However since early hours this morning every subdomain that would come under that wildcard is giving CF error 522

What is even stranger is that now the traffic spike between 9am-10:30am is giving NOERROR

I’m not sure how this is possible given the CF 522 errors?

To add to that my worker isn’t getting the additional traffic as the number of requests has barely gone up

Any ideas what I should try next?

[sjr]

@jack.t

What is even stranger is that now the traffic spike between 9am-10:30am is giving NOERROR

Because of your wildcard, the previous NXDOMAIN are now NOERROR as now they resolve. That’s for the DNS only.

Did you check your script with a random hostname yourself to see if that gives 522 as well?

[jack.t]

Yeah, testing this morning with a random hostname is how I discovered the 522 errors.

[jack.t]

Upon reviewing my DNS record can you see anything wrong with this?

I have set a CNAME * to point to the worker url

From what I have read this may not be the right way to point a CNAME to a worker?