Intermittent 403 errors

firewall
#1

Hi all,
for the last 24 hours we’ve been getting intermittent 403 errors, but since they are intermittent (and I’m just a php dev), it;s very hard to diagnose, so I need all the help I can get.
I’ll try to explain the situation…

We have a single web server (cssigniter.com). DNS, WAF, etc are on Cloudflare, and use a 3rd party cdn, KeyCDN. While this happens with all cdn zones, I’ll focus on one.
KeyCDN provides a url for the zone, (previewcdn-2246.kxcdn.com) which in order to use with ssl needs to have a CNAME record in cloudflare (preview.cssigniter.com).
We use this zone to serve static files, and all has been going well for the past 6 months until yesterday.
Assets would suddenly fail to load with a 403, and visiting an asset’s URL directly (while the problem “is active”) would result in a cloudflare challenge page. It doesn’t matter if I completed the challenge succesfully. I’d still get the same challenge page and never see the resource. A few minutes late, if I simply refresh the browser window, the resource appears. This whole thing repeats at (seemingly) random intervals.

I have already contacted KeyCDN support and they say their edge servers receive 403, and our host checked the LiteSpeed access logs and says there aren’t any related 403 records.

As a whole example, on this URL: https://www.cssigniter.com/preview/resto/the-restaurant-gallery/

At 11:26am, assets from this test url / zone returned 403.

Visiting the url of a single asset, e.g. https://preview.cssigniter.com/preview/resto/files/2014/10/Fotolia_43979273_Subscription_XXL-260x165.jpg presented a challenge page.

KeyCDN’s Speed test which shows requests and screenshot for the specific asset: https://tools.keycdn.com/speed?h=5cada9c20a23471fb8319782
Ray ID (not sure what this is): 4c535c8c1f3d9d5c

The prompt to solve the captcha wouldn’t end. I’ve solved it at least 5 times.
Refreshing didn’t help.
I waited a few minutes.
At 11:33, I refreshed the website tab. Everything is served properly.

At 11:56, I retried refreshing the page. Some are 200, some are 403.

I tried searching for this ray id on the CF firewall event log, no results.
I can’t whitelist KeyCDN’s servers as they don’t disclose any IPs.

I’m really lost!

Please help!

#2

I’m trying everything (normal) with cache disabled and not getting 403’s.

Are the 403’s displayed from the origin or are you seeing Cloudflare’s 1020 errors? If it’s 403’s from the origin, you’ll need to check those logs

Fixinfg Error 1020

#3

That’s the thing, it’s intermittent with no observable patterns (yet). It may happen in chrome but not in firefox, or vice versa. Or both. Or none.

The 403’s are cloudflare branded, and not really seeing anything 1020 related.

BUT: It just happened again, right after I purged the CDN zone. Maybe it has something to do with it?
I got to grab the headers (if useful):

  • General
  • Response Headers
    • cache-control: max-age=2
    • cf-chl-bypass: 1
    • cf-ray: 4c55bfc2fc079c35-AMS
    • content-encoding: gzip
    • content-type: text/html; charset=UTF-8
    • date: Wed, 10 Apr 2019 15:28:12 GMT
    • expect-ct: max-age=604800, report-uri=“https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct
    • expires: Wed, 10 Apr 2019 15:28:12 GMT
    • server: keycdn-engine
    • status: 403
    • vary: Accept-Encoding
    • x-frame-options: SAMEORIGIN
  • Request Headers
    • :authority: preview.cssigniter.com
    • :method: GET
    • :path: /preview/resto/files/2014/10/Fotolia_43979273_Subscription_XXL-945x500.jpg
    • :scheme: https
    • accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3
    • accept-encoding: gzip, deflate, br
    • accept-language: en-US,en;q=0.9,el;q=0.8
    • cache-control: no-cache
    • cookie: __cfduid=d8067abbfd054e1822b993b454831ea331554908998; _ga=GA1.2.468838263.1554909002; _gid=GA1.2.226227947.1554909002
    • pragma: no-cache
    • upgrade-insecure-requests: 1
    • user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36
#4

I would recommend opening a ticket with support directly, however in general putting a CDN in front of Cloudflare is not a configuration I recommend to my customers/ partners. Or if doing that you likely want to exclude static content from the WAF as a single user/IP triggering a WAF error can result in the CDN caching a failed response which could have significant splash damage to subsequent users requesting the same asset from the CDN.

You may also want to coordinate with the zone owner on Cloudflare who can look at Cloudflare’s security logs for potential additional information on specific requests.

1 Like
#5

Thank you,
I will do that right away!