Response headers are different between browser and online headers test

#1

Hi

I have installed the html-edge-cache worker from the worker examples GitHub repository and I am getting pretty low cache hits. When I view the headers from my Chrome browser I can see the header cf-cache-status & x-html-edge-cache-status but when I use an online tool like https://websniffer.cc/ these headers no longer exists even if I force the user agent to be my current browser.

Why is cloud flare returning separate headers? I have tried a few online tools and none are returning the same headers I can see in the browser.

Any thoughts?

Thanks in advance.

#2

Caching works by collocation users tend to hit always the same datacenters, but this could change for many reasons like load factors.

The test suite you are using might be hitting different datacenters.

Can you post the github template you are using?

#3

Hi, I am using this template https://github.com/Cloudflare/worker-examples/tree/master/examples/edge-cache-html along with the Wordpress plugin detailed on the page. It was created by [pmeenan]

I understand hitting different data centres can yield separate results but with this plugin the header x-html-edge-cache-status is always sent but I am not getting it at all when using rebot for example

redbot
HTTP/1.1 200 OK
Date: Sun, 14 Jul 2019 21:11:12 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=daf8822553b564c624c04bc0b2e2a368c1563138671;
expires=Mon, 13-Jul-20 21:11:11 GMT; path=/;
domain=.migraco.com; HttpOnly
Cache-Control: max-age=2633, public, must-revalidate, proxy-revalidate
CF-Ray: 4f667cd9cd165d70-BNE
Access-Control-Allow-Origin: *
Expect-CT: max-age=604800, report-uri=“https://report-
uri.Cloudflare.com/cdn-cgi/beacon/expect-ct”
Expires: Sun, 14 Jul 2019 21:55:06 GMT
Last-Modified: Sun, 14 Jul 2019 20:55:06 GMT
MS-Author-Via: DAV
Pragma: public
Public-Key-Pins: pin-sha256=""; pin-sha256=""; max-
age=31536000
Referrer-Policy:
Vary: Accept-Encoding,Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Powered-By: PleskLin
X-XSS-Protection: 1; mode=block
Server: Cloudflare

browser
access-control-allow-origin: *
age: 45
cache-control: max-age=2119, public, must-revalidate, proxy-revalidate
cf-cache-status: HIT
cf-ray: 4f668a8c48e5cc56-ZRH
content-encoding: br
content-type: text/html; charset=UTF-8
date: Sun, 14 Jul 2019 21:20:32 GMT
expect-ct: max-age=604800, report-uri=“https://report-uri.Cloudflare.com/cdn-cgi/beacon/expect-ct”
expires: Sun, 14 Jul 2019 21:55:06 GMT
last-modified: Sun, 14 Jul 2019 20:55:06 GMT
ms-author-via: DAV
pragma: public
public-key-pins: pin-sha256=""; pin-sha256=""; max-age=31536000
referrer-policy:
server: Cloudflare
status: 200
vary: Accept-Encoding,Cookie
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-html-edge-cache-status: Hit, Refreshed
x-html-edge-cache-version: 9
x-powered-by: PleskLin
x-xss-protection: 1; mode=block

#4

You need to change the header from the fetch. Then cache with the new header.
Ps: I am on the mobile hard to paste some code here.

#5

The headers from the fetch from the origin or the fetch from the cache or both?

That should fix that odd max-age number which I think is what you are referring to here.

But where are the other headers I am expecting?

#6

Look at this example.

#7

Thank, I can see where the my script is going wrong and I can fix that.

This does not explain why I am not seeing the other headers though and why the headers are different between my browser request and redbot request.

#8

Might be some weird stuff at redbot. Can you ask them? I have seen some test suites using lambda that not even cache DNS records.

#9

It’s not just redbot, hackertarget and websniffer all have the same results i.e. the edge cache headers are missing but when using the browser the header show.

Does Cloudflare do some sort of user agent detection and redirect the request?

In addition, do you know if it is possible to read the Browser TTL value set in the Page Rule so I can set that in Cache-Control to make it more flexible?

#10

Browser Cache Expiration set (30 min, 1 hour…) on the Dashboard prevails over the values set on the Workers, except when set to Respect Existing Headers.

Detection, there is a client trust score, not sure if that influences anything.

#11

Weirdly after debugging it is the server that is actually sending the odd cache control header. I am using W3 Total cache and it seems like it is counting down from 3600 which is the original setting

max-age=2225, public, must-revalidate, proxy-revalidate
max-age=2124, public, must-revalidate, proxy-revalidate
max-age=1595, public, must-revalidate, proxy-revalidate
max-age=1564, public, must-revalidate, proxy-revalidate

Not sure why this is but it is definitely coming from the response from the server.

#12

If anyone is curious…

The max-age countdown being sent from the origin is due to either my WP set up, W3 Total Cache or a combination of both. Its easy to reset it to a static number within the worker at cache time but its not obvious why it is counting down to 0.

The reason why the headers were missing is due to a race condition when getting the cache version stored in the KV with async and await and storing or retrieving the cached response from the caches.default. The promise was not resolving and the error (adding an unresolved promise to the headers) corrupts the headers.

I managed to solve the latter by querying the KV directly more often but this causes a performance hit of about 50ms per request, most probably down to my poor coding as well but using the KV in workers does have a cost.