All pages except main domain are bypassed by CF to origin

Website, Application, Performance Getting Started
1

What is the name of the domain?

zarqunpremium.com

What is the issue you’re encountering

Its quite complex, I am using Varnish on my origin with nginx as reverse proxy. Varnish is working on port 80 HTTP, and although its getting the request when main domain is opened, however varnish is not getting any request when opening any other pages. Don’t know why. Maybe CF requests port 80 for the main domain, which then Varnish gets it and processes it but for other all pages, CF is requesting port 443 and that is why its directly fetching from Nginx rather than Varnish. I don’t know how to debug this issue and make sure all requests get to Varnish/port 80 is this is the issue.

What are the steps to reproduce the issue?

Open the domain and see ‘X-Cache’ header is available which means Varnish received it. However all other pages such as /collections miss this header. I have also checked Varnish logs and there is no incoming request except “/”.

2

You should start by making sure Varnish is listening on port 443 and has a valid SSL certificate for your domain, otherwise your site will be insecure.

Cloudflare should never connect to your origin on port 80.

3

This is how I am starting Varnish.

ExecStart=/usr/sbin/varnishd \
          -a :80 \
          -a localhost:443,PROXY \
          -T localhost:6082 \
          -p feature=+http2 \
          -f /etc/varnish/default.vcl \
          -S /etc/varnish/secret \
          -s malloc,3g

And in my nginx configuration I am listening on port 443 for SSL.

4

I’m not entirely sure how that should be set up correctly, apparently Varnish does not support SSL itself.

Common setups seem to use Nginx for SSL termination, which proxies the requests to Varnish (via localhost), which proxies the requests back to Nginx again (via localhost).

In your setup, Varnish seems to be called directly from the outside.

5

Hi, I have a bit complicated thing going on here and have gotten myself very confused. I am trying to achieve varnish cache with nginx as reverse proxy. Currently, I am able to get Varnish perfectly working only on the home page i.e. "/", but nay other page doesn’t pass through Varnish at all.

After hours of struggling and a lot of help from Feryn from varnish I believe it’s due to my Nginx configuration somehow.

on Cloudflare I have 1 page rule, that forwards all requests to https://www.domain... suppose if user opened https://domain then should be redirected with www.

Screenshot from 2024-08-21 17-01-35
Screenshot from 2024-08-21 17-01-351024×542 38.1 KB

Nginx Configuration:

upstream domain_backend {
	server  unix:/run/php-fpm/domain.sock;
}

map $args $remove_noscript {
    ~PageSpeed=noscript $uri;
}

include /etc/nginx/bot.conf;

server {
  listen  8084;
  server_name www.domain.com domain.com;
  pagespeed on;
  pagespeed RespectXForwardedProto on;
  pagespeed ModifyCachingHeaders off;
  add_header X-Content-Type-Options nosniff;
  gzip on;

  location /render {
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Real-IP  $remote_addr;

        proxy_hide_header Cache-Control;
        add_header Cache-Control "private,max-age=600,must-revalidate";

        resolver 8.8.8.8 8.8.4.4;

        proxy_pass http://127.0.0.1:3000;
        rewrite .* /render/$scheme://$host$request_uri? break;
  }

  location / {
      index index.php index.html;
   	  if ($remove_noscript) {
          rewrite ^ $remove_noscript? permanent;
	     }

	     if ($is_bot = 1) {
          rewrite (.*) /render last;
	     }

	     try_files $uri $uri/ /index.php$is_args$args;
  }

  access_log /var/log/nginx/domain.com.access.log;
  error_log /var/log/nginx/domain.com.error.log;

  set $MAGE_ROOT /var/www/shared/domain/m2;
  include /var/www/shared/domain/m2/nginx.conf.sample;


#  return 301 https://www.domain.com$request_uri;
}

server {
  listen [::]:443; 
  listen  443 ssl http2;
  server_name www.domain.com domain.com;
  pagespeed on;
  pagespeed RespectXForwardedProto on;
  pagespeed ModifyCachingHeaders off;
  gzip on;  

  ssl_certificate /etc/letsencrypt/live/www.domain.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/www.domain.com/privkey.pem;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
  ssl_session_tickets off;
  include /etc/letsencrypt/options-ssl-nginx.conf;

  #OCSP
  ssl_trusted_certificate  /etc/letsencrypt/live/www.domain.com/chain.pem;
  ssl_stapling on;
  ssl_stapling_verify on;

  add_header X-Content-Type-Options nosniff;


  location / {
    index index.php index.html;
    if ($remove_noscript) {
        rewrite ^ $remove_noscript? permanent;
    }

    if ($is_bot = 1) {
        rewrite (.*) /render last;
    }

    try_files $uri $uri/ /index.php$is_args$args;

    proxy_pass http://127.0.0.1;
    proxy_set_header   Host $host;
    proxy_set_header   X-Forwarded-Host $http_host;
    proxy_set_header   X-Real-IP $remote_addr;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   Ssl-Offloaded "1";
    proxy_set_header   X-Forwarded-Port 443;
    proxy_set_header   X-Forwarded-Proto https;
    proxy_set_header   X-Forwarded-Ssl on;
    proxy_buffer_size   128k;
    proxy_buffers      4 256k;
    proxy_busy_buffers_size 256k;
    proxy_read_timeout 600;
    proxy_http_version 1.1;
    fastcgi_buffer_size 64k;
    fastcgi_buffers    16 32k;
  }
 

  location /robots.txt {
	   alias /var/www/shared/domain/m2/pub/robots.txt;
  }
  location /sitemap.xml {
	   alias /var/www/shared/domain/m2/pub/sitemap.xml;
  }

  location /render {
	   proxy_set_header X-Forwarded-For $remote_addr;
	   proxy_set_header X-Real-IP  $remote_addr;
  
	   proxy_hide_header Cache-Control;
     add_header Cache-Control "private,max-age=600,must-revalidate";

	   resolver 8.8.8.8 8.8.4.4;
	
	   proxy_pass http://127.0.0.1:3000;
	   rewrite .* /render/$scheme://$host$request_uri? break;
  }

  access_log /var/log/nginx/domain.com.access.log;
  error_log /var/log/nginx/domain.com.error.log;
  
  set $MAGE_ROOT /var/www/shared/domain/m2;
  include /var/www/shared/domain/m2/nginx.conf.sample;
}

Varnish Configuration:

vcl 4.0;

import std;
include "./vars.vcl";
include "./pagespeed-requirement.vcl";

sub vcl_recv {
if (req.restarts > 0) {
    set req.hash_always_miss = true;
}

if (req.http.host ~ "www.domain1.com" || req.http.host ~ "domain1.com") {
    set req.backend_hint = domain1;
    return (pass);
} else if (req.http.host ~ "www.domain.com" || req.http.host ~ "domain.com") {
    set req.backend_hint = domain;
} else if (req.http.host ~ "career.domain3.com") {
    set req.backend_hint = career;
    return (pass);
} else {
    set req.backend_hint = default;
    return (pass);
}

if (req.url ~ "^/.well-known/") {
  return (pass);
}

# Remove the proxy header to mitigate the httpoxy vulnerability
# See https://httpoxy.org/    
unset req.http.proxy;


if (req.method == "PURGE") {
    if (client.ip !~ purge) {
        return (synth(405, "Method not allowed"));
    }
    
# capistrano-magento2 gem for purging old content from varnish during it's deploy routine.
    if (!req.http.X-Magento-Tags-Pattern && !req.http.X-Pool) {
        return (synth(400, "X-Magento-Tags-Pattern or X-Pool header required"));
    }
    if (req.http.X-Magento-Tags-Pattern) {
      ban("obj.http.X-Magento-Tags ~ " + req.http.X-Magento-Tags-Pattern);
    }
# ban everything to catch assets as well
    if (req.http.X-Magento-Tags-Pattern == ".*") {
      ban("req.url ~ .*");
    }
    if (req.http.X-Pool) {
      ban("obj.http.X-Pool ~ " + req.http.X-Pool);
    }
    return (synth(200, "Purged"));
}

if (req.http.X-Forwarded-Proto !~ "https") {
    set req.http.location = "https://" + req.http.host + req.url;
    return (synth(750, "Permanently moved"));
}

if (req.method != "GET" &&
    req.method != "HEAD" &&
    req.method != "PUT" &&
    req.method != "POST" &&
    req.method != "TRACE" &&
    req.method != "OPTIONS" &&
    req.method != "DELETE") {
      /* Non-RFC2616 or CONNECT which is weird. */
      return (pipe);
}

# We only deal with GET and HEAD by default
if (req.method != "GET" && req.method != "HEAD") {
    return (pass);
}

# Bypass customer, shopping cart, checkout
if (req.url ~ "/customer" || req.url ~ "/checkout" || req.url ~ "/payment" || req.url ~ "/reclaim" || req.url ~ "/search" || req.url ~ "/order-confirmation-summary") {
    return (pass);
}

# Bypass health check requests
if (req.url ~ "/health_check.php") {
    return (pass);
}

# Set initial grace period usage status
set req.http.grace = "none";

# normalize url in case of leading HTTP scheme and domain
set req.url = regsub(req.url, "^http[s]?://", "");

# collect all cookies
std.collect(req.http.Cookie);

# Compression filter. See https://www.varnish-cache.org/trac/wiki/FAQ/Compression
if (req.http.Accept-Encoding) {
    if (req.url ~ "\.(jpg|jpeg|webp|png|gif|gz|tgz|bz2|tbz|mp3|ogg|swf|flv|webp)$") {
        # No point in compressing these
        unset req.http.Accept-Encoding;
    } elsif (req.http.Accept-Encoding ~ "gzip") {
        set req.http.Accept-Encoding = "gzip";
    } elsif (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
        set req.http.Accept-Encoding = "deflate";
    } else {
        # unknown algorithm
        unset req.http.Accept-Encoding;
    }
}

# Remove all marketing get parameters to minimize the cache objects
if (req.url ~ "(\?|&)(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=") {
    set req.url = regsuball(req.url, "(gclid|cx|ie|cof|siteurl|zanpid|origin|fbclid|mc_[a-z]+|utm_[a-z]+|_bta_[a-z]+)=[-_A-z0-9+()%.]+&?", "");
    set req.url = regsub(req.url, "[?|&]+$", "");
}

# Static files caching
if (req.url ~ "^/(pub/)?(media|static)/") {
    # Static files should not be cached by default
    return (pass);

    # But if you use a few locales and don't use CDN you can enable caching static files by commenting previous line (#return (pass);) and uncommenting next 3 lines
    #unset req.http.Https;
    #unset req.http.X-Forwarded-Proto;
    #unset req.http.Cookie;
}

# Authenticated GraphQL requests should not be cached by default
if (req.url ~ "/graphql" && req.http.Authorization ~ "^Bearer") {
    return (pass);
}

return (hash);
}


sub vcl_hash {
    if ((req.url !~ "/graphql" || !req.http.X-Magento-Cache-Id) && req.http.cookie ~ "X-Magento-Vary=") {
    hash_data(regsub(req.http.cookie, "^.*?X-Magento-Vary=([^;]+);*.*$", "\1"));
}

# For multi site configurations to not cache each other's content
if (req.http.host) {
    hash_data(req.http.host);
} else {
    hash_data(server.ip);
}

# To make sure http users don't see ssl warning
if (req.http.X-Forwarded-Proto) {
    hash_data(req.http.X-Forwarded-Proto);
}

if (req.http.cookie ~ "currency=") {
set req.http.X-TMP = regsub(req.http.cookie, ".*currency=([^;]+);.*","\1");
hash_data(req.http.X-TMP);
unset req.http.X-TMP;
}

if (req.url ~ "/graphql") {
    call process_graphql_headers;
}
}

sub process_graphql_headers {
if (req.http.X-Magento-Cache-Id) {
    hash_data(req.http.X-Magento-Cache-Id);

    # When the frontend stops sending the auth token, make sure users stop getting results cached for logged-in users
    if (req.http.Authorization ~ "^Bearer") {
        hash_data("Authorized");
    }
}

if (req.http.Store) {
    hash_data(req.http.Store);
}
if (req.http.Content-Currency) {
    hash_data(req.http.Content-Currency);
}
}

sub vcl_backend_response {
set beresp.grace = 3d;

if (beresp.http.content-type ~ "text") {
    set beresp.do_esi = true;
}

if (bereq.url ~ "\.js$" || beresp.http.content-type ~ "text") {
    set beresp.do_gzip = true;
}

if (beresp.http.X-Magento-Debug) {
    set beresp.http.X-Magento-Cache-Control = beresp.http.Cache-Control;
}

# cache only successfully responses and 404s
if (beresp.status != 200 && beresp.status != 404) {
    set beresp.ttl = 0s;
    set beresp.uncacheable = true;
    return (deliver);
} elsif (beresp.http.Cache-Control ~ "private") {
    set beresp.uncacheable = true;
    set beresp.ttl = 86400s;
    return (deliver);
}

# validate if we need to cache it and prevent from setting cookie
if (beresp.ttl > 0s && (bereq.method == "GET" || bereq.method == "HEAD")) {
    unset beresp.http.set-cookie;
}

   # If page is not cacheable then bypass varnish for 2 minutes as Hit-For-Pass
   if (beresp.ttl <= 0s ||
       beresp.http.Surrogate-control ~ "no-store" ||
       (!beresp.http.Surrogate-Control &&
       beresp.http.Cache-Control ~ "no-cache|no-store") ||
       beresp.http.Vary == "*") {
        # Mark as Hit-For-Pass for the next 2 minutes
        set beresp.ttl = 120s;
        set beresp.uncacheable = true;
    }

return (deliver);
}

sub vcl_deliver {
if (resp.http.X-Magento-Debug) {
    if (resp.http.x-varnish ~ " ") {
        set resp.http.X-Magento-Cache-Debug = "HIT";
        set resp.http.Grace = req.http.grace;
    } else {
        set resp.http.X-Magento-Cache-Debug = "MISS";
    }
} else {
    unset resp.http.Age;
}

if (obj.hits > 0) {
    set resp.http.X-Cache = "HIT";
    set resp.http.X-Cache-Hits = obj.hits;
} else {
    set resp.http.X-Cache = "MISS";
}

# Not letting browser to cache non-static files.
if (resp.http.Cache-Control !~ "private" && req.url !~ "^/(pub/)?(media|static)/") {
    set resp.http.Pragma = "no-cache";
    set resp.http.Expires = "-1";
    set resp.http.Cache-Control = "no-store, no-cache, must-revalidate, max-age=0";
}

unset resp.http.X-Magento-Debug;
unset resp.http.X-Magento-Tags;
unset resp.http.X-Powered-By;
unset resp.http.Server;
unset resp.http.X-Varnish;
unset resp.http.Via;
unset resp.http.Link;
}

sub vcl_synth {
if (resp.status == 750) {
    set resp.http.location = req.http.location;
    set resp.status = 301;
    return (deliver);
}
}

sub vcl_hit {
if (obj.ttl >= 0s) {
    # Hit within TTL period
    return (deliver);
}
if (std.healthy(req.backend_hint)) {
    if (obj.ttl + 300s > 0s) {
        # Hit after TTL expiration, but within grace period
        set req.http.grace = "normal (healthy server)";
        return (deliver);
    } else {
        # Hit after TTL and grace expiration
        return (restart);
    }
} else {
    # server is not healthy, retrieve from cache
    set req.http.grace = "unlimited (unhealthy server)";
    return (deliver);
}
}

I believe it has to do something with the HTTP & HTTPS Nginx server blocks and the fact that somehow how come only the home page works fine and nothing else, whereas isn’t Cloudflare passing traffic in the same manner? whether its home page or other pages.

6

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.