Missing traffic and 5xx errors

What is the name of the domain?

iplan.co.il

What is the issue you’re encountering

Some of the proxied traffic doesn’t seem to be reaching my servers despite a caching rule exception. And some of that missing traffic is encountering 5xx errors

What steps have you taken to resolve the issue?

I have an endpoint, used mostly by facebook, to redirect its bot to an image on s3.
I set a cache-rule not to cache this endpoint so so facebook will always be redirected to the most recent image.

Despite that, when making API requests that should result in ~300 requests, I only see ~100 in my logs.
In addition, facebook reports that some of the requests fail with 500, 502, or 520. I can clearly see that all the responses I handle are 302 as expected.
If manually go to the endpoint the relevant response headers are:
alt-svc: h3=“:443”; ma=86400
cache-control: no-cache
cf-cache-status: DYNAMIC
cf-ray: 8d92d98cf810e183-MRS

Trying to reproduce myself has so far failed to produce results, a naive load testing at rates one or two order of magnitudes above what we see in practice worked without issues

For crying out loud - when trying to submit this issue I got an error saying I can’t include 4 links even though there was only one (in the headers) and it deleted all my content.

Trying again to include blocked headers for completeness’ sake:

content-type: image/png; charset=utf-8
date: Sun, 27 Oct 2024 12:57:04 GMT
location: ...
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
priority: u=0,i
referrer-policy: strict-origin-when-cross-origin
report-to: {"endpoints":[{"url":"..."}],"group":"cf-nel","max_age":604800}
server: cloudflare
server-timing: cfCacheStatus;desc="DYNAMIC"
server-timing: cfExtPri
server-timing: cfHdrFlush;dur=0
set-cookie: ...
status: 302 Found
x-content-type-options: nosniff
x-download-options: noopen
x-frame-options:
SAMEORIGIN x-permitted-cross-domain-policies:
none
x-request-id: dec3621b-5c88-4037-9653-cb732e1c8239
x-runtime: 0.053201
x-xss-protection:1; mode=block

May I ask what steps for troubleshooting have you tried already at your origin host/server? :thinking:

Do they still show when you temporary Pause Cloudflare? :thinking:
Is your Website working fine over HTTPS when paused?

If there was something challenged or blocked, however I wonder if you could check the Security → Events at Cloudflare dashboard under your Cloudflare account for your zone.

You should be able to see the challenged or blocked event under the Security tab → Events at Cloudflare dashboard for your zone and know exactly which security option was triggered. Could be Bot Fight Mode or Browser Integrity. I’d suggest to try find one with filter by entering your origin host/server IP address.

Which would be the reason the requests didn’t pass to the origin.

Otherwise, are you using Rate Limiting at Cloudflare?

May I ask what steps for troubleshooting have you tried already at your origin host/server? :thinking:

Mostly checking and cross-checking logs, Verifying that any other request is properly logged in HAProxy (which is the first level and logs everything), Verifying that nothing is blocking such traffic and that it wasn’t enough to trigger ddos protection (by triggering it manually and seeing that it does indeed require much higher traffic)

Do they still show when you temporary Pause Cloudflare? :thinking:
Is your Website working fine over HTTPS when paused?

As mentioned, I can’t actually reproduce this. It just happens for a while every few weeks.

If there was something challenged or blocked, however I wonder if you could check the Security → Events at Cloudflare dashboard under your Cloudflare account for your zone.

You should be able to see the challenged or blocked event under the Security tab → Events at Cloudflare dashboard for your zone and know exactly which security option was triggered. Could be Bot Fight Mode or Browser Integrity. I’d suggest to try find one with filter by entering your origin host/server IP address.

Which would be the reason the requests didn’t pass to the origin.

Yes, I tried to check for these. The events I see don’t line up with the traffic I’m missing - it’s all from Vietnam, TOR, etc. facebook ips are all conveniently have the vanity address of “face:b00c” and thus easily identifiable - they definitely don’t appear there.

Nor would a security event explain the traffic that I’m not seeing but is succeeding.

Otherwise, are you using Rate Limiting at Cloudflare?

I am not.