APO bypassing cache for googlebot/crawlers?

Can anyone confirm this information?

As mentioned in this topic, it looks like APO doens’t cache requests from crawlers.

When i disable the old “Cache Everything” Page Rule, my server gets overloaded.

The headers are clean: no cookies, no cache control/pragma, and no “Vary” either. Actually, there is a “Vary: accept-encoding”, but that header is from Cloudflare.

If i access my page from incognito, i get a page hit.
If i access logged in, i get a “cookie” Dynamic.

So looks like the cache is working for humans. But i think it isn’t the case for crawlers…

Running more tests.

  • When making requests using curl, the x-via header always returns “origin, bypass
  • When i use a force refresh in chrome, it returns “origin, no-cache”

I’ve also tried using services to check the headers, they all have the same result as using curl.

Anyone else experiencing this issue?

Do your CURL request headers comply with the stated rules for the APO e.g,
Request doesn’t contain any of the following headers:

  • “Cache-Control: no-cache”
  • “Cache-Control: private”
  • “Pragma:no-cache”
  • “Vary: *”

and is a GET request?

You might also try sending an accept: text/html header, it may require that too.

Cloudflare really needs to just provide the source for the worker for APO so we call all see what is going on, it is frustrating that they haven’t .

2 Likes

Hi there,

This is the curl command that i’m using:

curl -svo /dev/null

It’s a simple GET request, just to check the headers.

I’m sorry, i’m not an advanced user in terminal, so i don’t know how to pass specific header. What i do know is that i’m getting the same results using external services to check the headers, too.

I’ve been trying to get a response from Cloudflare about this issue, but no one gives me a proper answer. I sent a very detailed message to the support, and they gave me a one line generic response, stating that the high load (when i disable the Cache Everything rule) doesn’t mean that APO is not delivering cached content.

I would really appreciate if someone could give me some directions.

Just so everyone can check for themselves, this is my website: https://tecnoblog.net

I am also having the same problem. I don’t have any idea if APO is working correctly on my website (https://www.teacherph.com/).

I tried using curl (curl -I https://www.teacherph.com/) and this was the result:

x-via: speedwp/origin,bypass

Update:

curl -skD - -H “Accept-Encoding:gzip” -H “Accept: text/html” https://www.teacherph.com -o /dev/null
HTTP/1.1 200 OK
Date: Fri, 09 Oct 2020 00:06:58 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=ddf81a732e8c0c82d0a43cf6230665d701602202018; expires=Sun, 08-Nov-20 00:06:58 GMT; path=/; domain=.teacherph.com; HttpOnly; SameSite=Lax; Secure
CF-Ray: 5df3dbd6cf5018e8-SIN
Age: 2971
Cache-Control: max-age=0
Expires: Thu, 08 Oct 2020 23:10:13 GMT
Last-Modified: Thu, 08 Oct 2020 23:10:13 GMT
Link: https://www.teacherph.com/wp-json/; rel=“https://api.w.org/
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Vary: Accept-Encoding
CF-Cache-Status: HIT
cf-request-id: 05ac47ba3f000018e836330200000001
Expect-CT: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
X-Cache: HIT
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
x-via: speedwp/cache
X-XSS-Protection: 1; mode=block
Report-To: {“endpoints”:[{“url”:“https://a.nel.cloudflare.com/report?lkg-colo=35&lkg-time=1602202018"}],“group”:“cf-nel”,"max_age”:604800}
NEL: {“report_to”:“cf-nel”,“max_age”:604800}
Server: cloudflare
Content-Encoding: gzip

looks like WP APO is working for that request

Hi Sir!

How can I know if Cloudflare APO is really working on my site? I checked the headers on Chrome but the cache status is still miss and bypass.

This topic was automatically closed 5 days after the last reply. New replies are no longer allowed.