Thanks @judahgabriel.
I think @arunesh90 answer is pretty accurate.
If node-fetch only supports HTTP/1.1 then it should just try that version and succeed. When you see a status code, it means that the HTTP handshake succeeded and that there was an application error.
I took the example glitch, and simplified it further by making a local node project with the following index.js contents of index.js, I get a 200 response back and all the payload that we would expect.
const fetch = require('node-fetch');
async function doIt() {
const response = await fetch('https://sky.shiiyu.moe/');
console.log(response.headers);
}
doIt();
then
$ node index.js
Headers {
[Symbol(map)]: [Object: null prototype] {
date: [ 'Mon, 28 Jun 2021 23:47:55 GMT' ],
'content-type': [ 'text/html; charset=utf-8' ],
'transfer-encoding': [ 'chunked' ],
connection: [ 'close' ],
vary: [ 'Accept-Encoding' ],
'x-powered-by': [ 'Express' ],
'x-cluster-id': [ 'w3' ],
'x-process-time': [ '6' ],
'cf-cache-status': [ 'DYNAMIC' ],
'cf-request-id': [ '0af69ecedb00005428a4b21000000001' ],
'expect-ct': [
'max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"'
],
'report-to': [
'{"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v2?s=pMNsOtQhByFRsM%2FuVFesktyqqgRgaNKk%2ByVRegJSSny8fbKz8QvnA7RGvODXc7U%2BOqnuxaDYbq1XXNnlVqWmGW3Rlu4YN6mWOdRu6%2FjEiIcHzswEoiEynrqmiNo%3D"}],"group":"cf-nel","max_age":604800}'
],
nel: [ '{"report_to":"cf-nel","max_age":604800}' ],
server: [ 'cloudflare' ],
'cf-ray': [ '666acd915a515428-LHR' ],
'content-encoding': [ 'gzip' ],
'alt-svc': [
'h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400, h3=":443"; ma=86400'
]
}
}
Based on this trace it’s likely that the Cloudflare mitigations like bot detection are kicking in due to the circumstance around how PWABuilder decides to fetch content. Different mitigations can lead to different kind of responses, which could explain the differences that users report.
Asking users to disable protections avoids the problem but doesn’t seem ideal if it can be avoided. It would be better if Cloudflare trusted the fetches from PWABuilder but that is an area beyond my expertise. As far as I understand the architecture, a human sits in the loop and triggers a centralised service to fetch the assets and do the bundling etc. An alternative might be to fetch the assets in the user’s browser and then post them to the centralised service for further processing.
