Upcoming Change To Accept Encoding Header

Hi Cloudflare Community!

I wanted to inform you of an exciting upcoming change to Cloudflare’s Accept-Encoding header. This change will enable Cloudflare to support the transmission of Brotli compressed data from your origin servers.

Currently, Cloudflare forwards all HTTP requests to the origin with the following header:
Accept-Encoding: gzip

Over the coming weeks, we will be transitioning to the following header to allow the origin to send Brotli compressed content to Cloudflare:
Accept-Encoding: gzip, br

Impact of the Change

No action should be required. If you wish to disable Brotli at your origin or do not have Brotli support enabled, Cloudflare will continue to support Gzip and uncompressed content as usual.

If your origin supports Brotli encoding, you will be able to take advantage of its high compression ratios when transferring data to Cloudflare.

What is Brotli?

Brotli is a highly efficient data compression algorithm developed by Google. It offers high compression ratios and fast decompression speeds, making it particularly suitable for compressing text-based files such as HTML, CSS, and JavaScript. Since 2019, Brotli has been widely supported by web browsers. Cloudflare has been offering Brotli compression to end users customers since 2017, resulting in improved website performance and smaller file sizes. We are now extending support to origin servers.

Thanks,

Matt Bullock
FL Product Manager

17 Likes

This is fantastic!!

Will Cloudflare still decompress and then recompress when serving that data to eyeballs? For example, if I serve a highly compressed JS file with Brotli to Cloudflare, will my highly compressed version also be delivered to the user, or will Cloudflare decompress and the recompress at a lower compression ratio?

1 Like

I’d assume they have to decompress the file, in order to apply the usual proxy features. Plus, they need to take the acceptable client encoding into account.

2 Likes

@cherryjimbo we won’t unless, as @sandro mentioned, we need to apply proxy features. That’s it, if you disable all HTML-based features (ex: email obfuscation) and send us brotli-compressed HTML, we will keep the same compression strategy you picked!

1 Like

Does this mean Cloudflare would cache based on the compression? If I send br compressed data and Cloudflare caches that, that would not work with a client which does not support br.

2 Likes

this behavior won’t change from the one you can currently use. Right now, if client is not requesting brotli, we will not serve them brotli-compressed responses.

2 Likes

Yeah I assume it wouldn’t change from the current setup, just wondering about Cloudflare keeping the compression.

I would imagine that only works for content where Cloudflare was instructed not to cache, the proxy is not doing anything, and when the client accepted br as well, right?

To maintain the same compression as what the origin sends (i.e., Brotli Compression Level 11), the following proxy features should be disabled:

  • Email Obfuscation
  • Rocket Loader
  • Server Side Excludes (SSE)
  • Mirage
  • HTML Minification - JS and CSS can be left enabled.
  • Automatic HTTPS Rewrites (Not to be confused with always use HTTPS)

The API calls / BASH Script can be found below. Note that the below will disable all Minification features (HTML, CSS, JS), with only HTML requiring to be disabled.

If any of these features are enabled, your origin can still send Brotli Compression level 11. However, we will decompress, apply the feature(s), and recompress on the fly.

For browsers that do not accept Brotli Compression, we will decompress and send GZIP Compressed or Uncompressed.

#!/bin/bash

# Cloudflare API credentials
api_token="R3d4ct5dXXXXXXX"
email="[email protected]"
zone_id="c0779e7XXXXXXXXXXXXXXX6975e2fb9b"


# Disable Email Obfuscation
curl -s -X PATCH "https://api.cloudflare.com/client/v4/zones/$zone_id/settings/email_obfuscation" \
  -H "X-Auth-Email: $email" \
  -H "X-Auth-Key: $api_token" \
  -H "Content-Type: application/json" \
  --data '{"value": "off"}'

# Disable Rocket Loader
curl -s -X PATCH "https://api.cloudflare.com/client/v4/zones/$zone_id/settings/rocket_loader" \
  -H "X-Auth-Email: $email" \
  -H "X-Auth-Key: $api_token" \
  -H "Content-Type: application/json" \
  --data '{"value": "off"}'

# Disable Server Side Excludes (SSE)
curl -s -X PATCH "https://api.cloudflare.com/client/v4/zones/$zone_id/settings/server_side_exclude" \
  -H "X-Auth-Email: $email" \
  -H "X-Auth-Key: $api_token" \
  -H "Content-Type: application/json" \
  --data '{"value": "off"}'

# Disable Mirage
curl -s -X PATCH "https://api.cloudflare.com/client/v4/zones/$zone_id/settings/mirage" \
  -H "X-Auth-Email: $email" \
  -H "X-Auth-Key: $api_token" \
  -H "Content-Type: application/json" \
  --data '{"value": "off"}'

# Disable HTML Minification NOTE: you only need to disable HTML this call disables all
curl -s -X PATCH "https://api.cloudflare.com/client/v4/zones/$zone_id/settings/minify" \
  -H "X-Auth-Email: $email" \
  -H "X-Auth-Key: $api_token" \
  -H "Content-Type: application/json" \
  --data '{"value":{"js":"off","css":"off","html":"off"}}'

# Disable Automatic HTTPS Rewrites - NOT to be confused with Always use HTTPS
curl -s -X PATCH "https://api.cloudflare.com/client/v4/zones/$zone_id/settings/automatic_https_rewrites" \
  -H "X-Auth-Email: $email" \
  -H "X-Auth-Key: $api_token" \
  -H "Content-Type: application/json" \
  --data '{"value": "off"}'
2 Likes

All right, so essentially one only needs to disable all features which possibly change the content. Caching itself is fair and Cloudflare will automatically recompress as gz if a client requested a br cached file but does not support br.

4 Likes

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.

@mbullock this doesn’t seem to be the case - when sending a brotli level 6 encoded response with no-transform, the served page size (text/html) is 10% lower than when not using no-transform.

Cloudflare uncompresses and recompresses, even though it shouldn’t. I have all of the above things deactivated.

Can you reproduce this issue on your end or is there anything I can provide to showcase the issue?

EDIT: just checked: the Accept-Encoding header for the origin request has “gzip” only, there’s no “br” in there, which should be there according to this announcement?

1 Like

Just popping in here to say that I have a wee bit of content on origin that is brotli pre-compressed

However I am also seeing that CF only sends Accept-Encoding: gzip to origin and thus I need to decompress this content in order to serve back to origin.

Is this intentional?

Hi,

We had to disable this again due to an issue in Novemember.

We will be looking to retry this later this quarter. Sorry for the delay!

Matt

2 Likes

It would be appreciated if the disabling of this feature could be added to:

  • The documentation; https://developers.cloudflare.com/speed/optimization/content/brotli/content-compression/#content-compression-from-origin-servers-to-the-cloudflare-network
  • The blog post; All the way up to 11: Serve Brotli from origin and Introducing Compression Rules https://blog.cloudflare.com/this-is-brotli-from-origin/

I just spent a couple of hours debugging and experimenting before I found this forum post. While I learned some interesting things, others could be prevented the same journey.

Sidenote: I have noticed that Cloudflare responds to repeated requests for the same static no-cache resource with Brotli responses that differ in Content-Length though they decompress to the exact same thing. Do you happen to know the reason behind this?

4 Likes

We have updated our Developers Documentation with the caveat. Thanks for raising.

1 Like

Great news everyone we are beginning to roll this out to self serve plans this week 26th February 2024 - Starting with Free Plans.

Thanks,

Matt

5 Likes

This has been enabled for all Free Plans since 1st March 2024.
This week we will be enabling for all other self serve customers.

Again let me know if any issues!

Matt

8 Likes

A Fri-yay update.

All Free Plans, Pro Plans and Business Plans have the new Accept Encoding header has been enabled, Since yesterday - 7th March 2024.

We will begin Enterprise Plans from the 18th March 2024.

Any questions let me know!

Matt

2 Likes