HTML fetch working in preview but "too many redirects" error while live

I have a worker with the following code, which is almost the same as the fetch html example:

const url = "https://www.euskadi.eus/hasiera/"

/**
 * gatherResponse awaits and returns a response body as a string.
 * Use await gatherResponse(..) in an async function to get the response body
 * @param {Response} response
 */
async function gatherResponse(response) {
  const { headers } = response
  const contentType = headers.get("content-type") || ""
  if (contentType.includes("application/json")) {
    return JSON.stringify(await response.json())
  }
  else if (contentType.includes("application/text")) {
    return await response.text()
  }
  else if (contentType.includes("text/html")) {
    return await response.text()
  }
  else {
    return await response.text()
  }
}

async function handleRequest() {
  const init = {
    headers: {
      "content-type": "text/html;charset=UTF-8",
    }
  }
  const response = await fetch(url, init)
  const results = await gatherResponse(response)
  return new Response(results, init)
}

addEventListener("fetch", event => {
  return event.respondWith(handleRequest())
})

The only difference is the value of the url variable.

This code runs well while in preview, returning the HTML properly:

However, opening this in a browser will return a 1101 error (Too many redirects, according to the logs).

Debugging this, I added the option redirect: "manual" in the init object. After doing that, preview still worked fine, but the browser page now returns a 301 page that tries to redirect to the same page (live code: https://dawn-mud-2bab.victor-martin84.workers.dev/)

This error only happens in the www.euskadi.eus domain. Any other url from other domains that I tested works well.

I guess there must be some configuration on their server colliding with the worker fetch requests, but why does it work on the preview? I tried to add some headers to the fetch request to avoid the redirect, but with no success.

Any idea on how to make this work? Thanks!

Hello! I’ve not been able to reproduce the redirect, but it is very likely the 301 response is coming from the origin server. The challenge is to determine what characteristic of the workers sub-request is causing the origin server to respond with a 301. I would suggest trying to recreate the sub-request with curl so that you can see how the origin respondes.

For example, creating a request with curl where you can see the HTTP response headers and adding a custom request header as well:

curl -vso /dev/null https://www.euskadi.eus/hasiera/ -H "content-type: text/html;charset=UTF-8"
*   Trying 217.124.154.78...
* TCP_NODELAY set
* Connected to www.euskadi.eus (217.124.154.78) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [229 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [61 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [6031 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [589 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=www.euskadi.eus
*  start date: Jul  8 10:07:55 2019 GMT
*  expire date: Jul  8 10:07:55 2021 GMT
*  subjectAltName: host "www.euskadi.eus" matched cert's "www.euskadi.eus"
*  issuer: C=ES; O=IZENPE S.A.; OU=AZZ Ziurtagiri publikoa - Certificado publico SCA; CN=EAEko Herri Administrazioen CA - CA AAPP Vascas (2)
*  SSL certificate verify ok.
> GET /hasiera/ HTTP/1.1
> Host: www.euskadi.eus
> User-Agent: curl/7.64.1
> Accept: */*
> content-type: text/html;charset=UTF-8
> 
< HTTP/1.1 200 OK
< Date: Tue, 08 Jun 2021 19:30:40 GMT
< Server: Apache
< Set-Cookie: r01euskadiUserCookie=YL-FYAJGD3X2qpL50bPSnwAAAMA; path=/; domain=euskadi.eus; expires=Wed, 08-Jun-2022 19:30:40 GMT
< Set-Cookie: r01euskadiCookie=web01_eu; path=/; domain=euskadi.eus; expires=Wed, 08-Jun-2022 19:30:40 GMT
< Set-Cookie: r01PortalInfo=web01-hasiera/eu; path=/; domain=www.euskadi.eus
< Access-Control-Allow-Origin: *
< Strict-Transport-Security: max-age=525600
< Vary: Cookie,User-Agent,X-Forwarded-Proto,isSecure
< ETag: "3a23-5c35e9e326c00"
< Accept-Ranges: bytes
< Transfer-Encoding: chunked
< Content-Type: text/html; charset=ISO-8859-1
< 
{ [655 bytes data]
* Connection #0 to host www.euskadi.eus left intact
* Closing connection 0