Cloudflare workers published on workers.dev unreachable - Error 523 to Error 522

#1

Hello –

Trying to test an existing and new cloud workers in the new workers dashboard, as we recently started receiving 522 Timeouts when deployed on a route.

Edits were done both ways – locally tested and deployed to workers.dev subdomains using wrangler CLI, as well as directly from the dashboard (https://dash.cloudflare.com/)

Waited 1+ hour relying on wrangler documentation referral to the initial wait time .

pushing a new workers.dev Worker project you may initially see 523 errors. Do not fear! The DNS is
propagating and can take a few seconds. It should work after a minute or so.

After about an hour of 523 Origin was still unreachable with the error switching to 522 Timeout
DOMAIN: https://xxxxxxx.workers.dev
Error 522 Ray ID: 536a640d42c9cf04 • 2019-11-16 15:10:23 UTC

Some new/additional info:
The workers are fully functional when tested on the dashboard (https://dash.cloudflare.com/), timeout (after 20 mins) occurs when tested externally via curl/Postman, and only in cases where the POST request contains data:

Error 1105 Ray ID: 537436e24bc1cedc • 2019-11-17 20:07:14 UTC

Temporarily unavailable

What happened?

You’ve requested a page on a website (our_domain) that is on the Cloudflare network. The page could not be rendered due to a temporary fault.

Cloudflare Ray ID: 537436e24bc1cedc • Your IP: xxx.xxx.xxx.xxx• Performance & security by Cloudflare

Any help/recommendation/tip will be greatly appreciated!

#2

Hi @vrateopsus, I’m not sure there’s enough information here to debug this, so I’d recommend filing a support ticket to investigate. You can do so by emailing [email protected] from the email address associated with your Cloudflare account.

One thing, though:

You’ve requested a page on a website (our_domain) that is on the Cloudflare network. The page could not be rendered due to a temporary fault.

Does your actual subdomain contain an underscore? Due to a bug, we accidentally allowed such subdomains, but they are non-conformant and thus rather buggy.

Harris

1 Like
#3

Thanks for your response Harris.

I will try to provide as much detail as I can – hoping that this helps others who might run into the same issue.

I am new to working with cloud workers, primarily borrowing code snippets from templates with minor customization.

I was able to isolate the issue to the following line in our worker code, adapted from templates:

if (error-condition)
return new Response(“Error message”, {status: 401})

Changing response to include headers from the origin (despite the fact that we are in fact trying to block the processing of the request here) seems to address the timeout problems:

if (error-condition) {
let errResponse = await fetch(request)
return new Response(“Error message”, {headers: errResponse.headers, status:401})
}

It appears there is something about the default headers generated for a new response by the worker deployed on a workers.dev subdomain.

Here are other points which helped me isolate the problem, and may give you some insights hopefully.

1 - The timeout happens only for POST requests with data posted in body. Same POST request without a body, returns a response immediately.

2 - This worker was functional until very recently (up to a week at most)

3 - There is no way to reproduce the 522 timeout on the worker path using the dashboard testing utility, this is only detected externally (curl/postman). PS: I discarded the cache, and have been debugging in Cloudflare dev mode.

PPS: No underscores in the subdomain.
– zeyno

#4

Hi @vrateopsus, that information helps, but there are still a lot of unknown variables. Are you able to share a minimal example script that reproduces the behavior?

#5

Here is a short script that stalls with an eventual 522 timeout:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

/**
 * Respond to the request
 * @param {Request} request
 */
async function handleRequest(request) {
  if (request.method == 'POST') {
      try {
        const secretkey = request.headers.get('Some-Secret-Key')
        if (!secretkey || secretkey === null) {
            return new Response("Secret key required in the header", {status:401})
        }
        let authorizedResponse = await fetch(request)
        authorizedResponse = new Response(authorizedResponse.body, authorizedResponse)
        return authorizedResponse
      } catch (e) {
	     return new Response("Worker Exception: " + e.message, {status: 500})
  	  }
  	}
    // No processing for all non-POST requests
    return await fetch(request)
}

Calling the worker through deployed path:

522 Timeout

curl --trace timeout.trace.txt --location --request POST “https://path-to-worker” --header “Content-Type: application/json” --data “{
“payload”: “hello”
}”

Without data – no timeout

curl --trace timeout.trace.txt --location --request POST “https://path-to-worker” --header “Content-Type: application/json”
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 33 100 33 0 0 111 0 --:–:-- --:–:-- --:–:-- 111Secret key required in the header

With header provided – no timeout, expected response from the original destination returned.

curl --trace timeout.trace.txt --location --request POST “https://path-to-worker” --header “Content-Type: application/json” --header “Some-Secret-Key:xxxxxxx”
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 23 100 23 0 0 46 0 --:–:-- --:–:-- --:–:-- 46{“message”:“Forbidden”}

Also for your reference, here is the version with response creation changed working fine:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

/**
 * Respond to the request
 * @param {Request} request
 */
async function handleRequest(request) {
  if (request.method == 'POST') {
      try {
        const secretkey = request.headers.get('Some-Secret-Key')
        if (!secretkey || secretkey === null) {
            let errResponse = await fetch(request)
            return new Response("Secret key required in the header", {headers: errResponse.headers, status:401})
        }
        let authorizedResponse = await fetch(request)
        authorizedResponse = new Response(authorizedResponse.body, authorizedResponse)
        return authorizedResponse
      } catch (e) {
	     return new Response("Worker Exception: " + e.message, {status: 500})
  	  }
  	}
    // No processing for all non-POST requests
    return await fetch(request)
}
#6

Hi @vrateopsus,

I tried to reproduce what you’re seeing, and I can’t. The script that expresses the behavior for you just returns a 401 with “Secret key required in the header” immediately. I.e., it seems to work fine.

One thing I noticed is that the biggest difference between the two scripts isn’t the response headers, but actually that the working script fully reads the request body (by means of the fetch(request)), whereas the broken one does not. This is relevant because when a worker returns a response, all further reading of the request body is ended, which can cause problems for some clients. However, I don’t see how that could cause a 522…

So, I’m a bit stumped. :frowning: I think the next step is to file a support ticket.

Harris

1 Like
#7

One thing I noticed is that the biggest difference between the two scripts isn’t the response headers, but actually that the working script fully reads the request body (by means of the fetch(request) ), whereas the broken one does not. This is relevant because when a worker returns a response, all further reading of the request body is ended, which can cause problems for some clients.

I am not sure how either, but this is indeed relevant – I just tested it, by adding a fetch(request) prior to returning a brand new response instance (so no response header change). And as you suggested there are no timeouts. Thanks for your help in getting some understanding of the issue.

PS: About reporting to support, there is considerable lag for ticket processing for free accounts – we have another (more urgent) one waiting in the queue for 2+ days – I will send an email to support with this thread.

#8

If the request bodies that you’re expecting aren’t too huge, a workaround might be to call await request.arrayBuffer() before returning the error response. Then you’re not bothering the origin server with the wasted subrequest.

However, that’s still suboptimal and shouldn’t theoretically be necessary, so it’d be good to get to the bottom of the issue. :confused:

1 Like