Performance issue when Workers fetch a HTTPS site

What’s happened

When Cloudflare fetch files from the HTTPS origin site the speed will slow down (compared to HTTP)

Recently, I found that users download uncached files through Cloudflare very slowly.
After investigating for a period of time, I found that when Cloudflare communicates with the origin site via HTTPS, the speed will be more than 10 times slower
(12MB/s HTTP → 300KB/s HTTPS)

Both Cloudflare workers and Cloudflare has this problem.

My question

How to avoid or resolve this problem?

Quick Look

Use worker to Fetch a HTTP site (12MB/s)

https://https-performance-issue.ikoa.workers.dev/?by=http
wget https://https-performance-issue.ikoa.workers.dev/?by=http -S -O /dev/null

Use worker to Fetch a HTTPS site (300KB/s)

https://https-performance-issue.ikoa.workers.dev/?by=https
wget https://https-performance-issue.ikoa.workers.dev/?by=https -S -O /dev/null

Reproduce

  1. start a caddy file-server
    Caddyfile

    http://www.199-19-224-99.sslip.io, https://www.199-19-224-99.sslip.io { 
        file_server
    }
    
  2. Deploy cf workers

    addEventListener("fetch", (event) => {
      event.respondWith(
        handleRequest(event.request).catch(
          (err) => new Response(err.stack, { status: 500 })
        )
      );
    });
    
    async function fetchFrom(request) {
        let response = await fetch(request);
        response = new Response(response.body, response);
        response.headers.set("from", request.url);
        return response;
    }
    
    async function handleRequest(request) {
      let url = new URL(request.url);
      if (url.searchParams.get("by") === "https") {
        let response = await fetchFrom(new Request("https://www.199-19-224-99.sslip.io/test.bin", request))
        return response; 
      } else {
        let response = await fetchFrom(new Request("http://www.199-19-224-99.sslip.io/test.bin", request))
        return response; 
      }
    } 
    
  3. wget it
    By HTTP (13.2MB/s):

    wget https://https-performance-issue.ikoa.workers.dev/?by=http -S -O /dev/null

    [email protected]:~# wget https://https-performance-issue.ikoa.workers.dev/?by=http -S -O /dev/null
    --2021-09-07 08:59:46--  https://https-performance-issue.ikoa.workers.dev/
    Resolving https-performance-issue.ikoa.workers.dev (https-performance-issue.ikoa.workers.dev)... 104.21.235.46, 104.21.235.45, 2606:4700:3038::6815:eb2d, ...
    Connecting to https-performance-issue.ikoa.workers.dev (https-performance-issue.ikoa.workers.dev)|104.21.235.46|:443... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 200 OK
      Date: Tue, 07 Sep 2021 08:59:46 GMT
      Content-Length: 1073741824
      Connection: keep-alive
      CF-Ray: 68aebe2dad592bf6-FRA
      ETag: "qz22dxhra0hs"
      From: http://www.199-19-224-99.sslip.io/test.bin
      Last-Modified: Tue, 07 Sep 2021 08:32:21 GMT
      Vary: Accept-Encoding
      CF-Cache-Status: MISS
      Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
      Report-To: {"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=EDtWfiraqsdTf%2FQngA5zDimnvmL3QbE2iEYoDngYZ05drpKPpfuMAQN8FVuMv2qOaHaoen2kbKydpo1CPPjLu4kMWjhQzjWKFFy2oux9vJ2Uyvmb913bxMt0sB6FmXP5AQ8pFfnSyXgu4i1lKY83ICp1QJhwCkA6kvrd"}],"group":"cf-nel","max_age":604800}
      NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
      Server: cloudflare
      alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
    Length: 1073741824 (1.0G)
    Saving to: ‘/dev/null’
    
    /dev/null             6%[====>               ]  62.44M  13.2MB/s    eta 87s  
    

    By HTTPS (207KB/s ):

    wget https://https-performance-issue.ikoa.workers.dev/?by=https -S -O /dev/null

    [email protected]:~# wget https://https-performance-issue.ikoa.workers.dev/?by=https -S -O /dev/null
    --2021-09-07 08:59:17--  https://https-performance-issue.ikoa.workers.dev/
    Resolving https-performance-issue.ikoa.workers.dev (https-performance-issue.ikoa.workers.dev)... 104.21.235.46, 104.21.235.45, 2606:4700:3038::6815:eb2d, ...
    Connecting to https-performance-issue.ikoa.workers.dev (https-performance-issue.ikoa.workers.dev)|104.21.235.46|:443... connected.
    HTTP request sent, awaiting response... 
      HTTP/1.1 200 OK
      Date: Tue, 07 Sep 2021 08:59:19 GMT
      Content-Length: 1073741824
      Connection: keep-alive
      CF-Ray: 68aebd7cc9ad073e-FRA
      ETag: "qz22dxhra0hs"
      From: https://www.199-19-224-99.sslip.io/test.bin
      Last-Modified: Tue, 07 Sep 2021 08:32:21 GMT
      Vary: Accept-Encoding
      CF-Cache-Status: MISS
      Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
      Report-To: {"endpoints":[{"url":"https:\\/\\/a.nel.cloudflare.com\\/report\\/v3?s=3VM8fXLDgZu4brb2TWdp9a1qe%2BpFoJnzJegLelR%2Bf8h%2F29eqLkWprAjTFG6H7PnQAqE8%2FmDQZ2SdvccsMip76WJL4qb0ZrLx6QmJSEqPZZI%2BsYXYjq1dDCNaiyGgPRd2JTw5ZDKEi1KKUUpJLvPU6F7AlUl27wVLiwAH"}],"group":"cf-nel","max_age":604800}
      NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
      Server: cloudflare
      alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400, h3-28=":443"; ma=86400, h3-27=":443"; ma=86400
    Length: 1073741824 (1.0G)
    Saving to: ‘/dev/null’
    
    /dev/null       0%[              ] 548.77K   207KB/s    
    

That’s quite interesting, I’ve not seen anyone mention this before when talking about wget speeds of workers, but could be explained by the additional CPU overhead of https?

It’s probably worth posting in the Cloudflare developer discord :slight_smile:

Feel free to join at Cloudflare Workers

Maybe…

But as far as I know, If I fetch a file from Backblaze(a company that provide object storage) S3 API, then that performance issue doesn’t exist

Could it caused by the TLS version?

Oh that’s really odd, be interesting to compare SSL key size as well as other possible variables.