4XX Erros @ Analytics (Beta)

Hi guys,

A few weeks ago I noticed that the new Analytics (Beta) is reporting a high amount of 4xx errors for one of the accounts I manage. Since the web server that hosts this website does not provide me logs, I installed the awesome @chasers Logflare app for the 2 domains belonging to this account.

As a Worker-powered app, Logflare can log all web requests/responses, so I was expecting to catch these 4xx errors, however logs don’t match data displayed in Cloudflare’s analytics tools.

PS: This post was originally created in order to report the strange amount of 4xx errors reported by Analytics Beta but not by Logflare, however, by collecting more accurate information and rewriting it I realized that the missing requests in the Logflare logs go far beyond those errors.

So I’ve collected as much information as I can and I’m sharing it below in order to clarify these issues.

Thanks in advance!


As this post got quite extensive, I structured it in the following way:

🔶 Section
|
|__#️⃣ Topic
   |
   |__🔹 Subtopic

I hope it is minimally understandable. Please let me know if I can improve it.


:large_orange_diamond: Comparison

:one: All requests

Comparing Logflare to Cloudflare, there are 4,883 missing requests (~9%).

Chart data
Cloudflare [#] Logflare [#]
7 Feb, 7:00 PM 2798 2553
7 Feb, 8:00 PM 2774 2536
7 Feb, 9:00 PM 3299 2997
7 Feb, 10:00 PM 2927 2694
7 Feb, 11:00 PM 3362 3069
8 Feb, 12:00 AM 3476 3180
8 Feb, 1:00 AM 3124 2845
8 Feb, 2:00 AM 2804 2544
8 Feb, 3:00 AM 1952 1778
8 Feb, 4:00 AM 1324 1222
8 Feb, 5:00 AM 1107 930
8 Feb, 6:00 AM 591 538
8 Feb, 7:00 AM 359 314
8 Feb, 8:00 AM 559 522
8 Feb, 9:00 AM 887 813
8 Feb, 10:00 AM 1380 1266
8 Feb, 11:00 AM 2072 1887
8 Feb, 12:00 PM 2561 2335
8 Feb, 1:00 PM 3242 2969
8 Feb, 2:00 PM 3224 2920
8 Feb, 3:00 PM 2814 2594
8 Feb, 4:00 PM 3436 3143
8 Feb, 5:00 PM 3060 2802
8 Feb, 6:00 PM 3153 2890
56285 51341

:two: Missing vs. Firewall

Facing the missing requests with the firewall events over time:

Chart data
Missing [#] Firewall [#]
7 Feb, 7:00 PM 247 2
7 Feb, 8:00 PM 234 6
7 Feb, 9:00 PM 303 27
7 Feb, 10:00 PM 226 0
7 Feb, 11:00 PM 291 1
8 Feb, 12:00 AM 300 3
8 Feb, 1:00 AM 275 12
8 Feb, 2:00 AM 256 4
8 Feb, 3:00 AM 172 1
8 Feb, 4:00 AM 98 1
8 Feb, 5:00 AM 160 52
8 Feb, 6:00 AM 53 5
8 Feb, 7:00 AM 45 1
8 Feb, 8:00 AM 35 2
8 Feb, 9:00 AM 74 7
8 Feb, 10:00 AM 114 3
8 Feb, 11:00 AM 183 11
8 Feb, 12:00 PM 225 13
8 Feb, 1:00 PM 271 2
8 Feb, 2:00 PM 300 27
8 Feb, 3:00 PM 216 2
8 Feb, 4:00 PM 287 3
8 Feb, 5:00 PM 258 10
8 Feb, 6:00 PM 260 6
4883 201

:three: Missings vs. 4xx Errors vs. Firewall

As the 4xx errors chart only displays the total value, this is the only possible correlation:

Chart data
Missing [#] 4xx errors [#] Firewall [#]
4883 910 201

:large_orange_diamond: Cloudflare data

:zero: Summary

Requests[1] 4xx errors Requests[2] Threats Firewall
(all hostnames) 54010 910
hostname.gtld 56285 2 201
hostname.cctld 6 0 2
54010 910 56291 2 203
TABLE INFO
  • Account Analytics
    • Requests[1]: Account → Analytics (Beta) → {Requests}
    • 4xx errors: Account → Analytics (Beta) → {4xx errors}
  • Site Analytics
    • Requests[2]: Zone → Analytics → Traffic → {Requests}
    • Threats: Zone → Analytics → Security → {Threats}
    • Firewall: Zone → Firewall → Overview → {Events}

PS-1: I forgot to make notes on the Threats data, so these are only available in this Summary.
Since there were only 2 events in this ranking, I don’t think this is a problem.

PS-2: The discrepancy between Requests[1] and Requests[2] metrics is addressed in:
🔶 Annotations → 1️⃣ Cloudflare → 🔹 Analytics: Account vs. Site.

:one: Account Analytics

The data that interest us are Requests (first of “General”) and 4xx errors (first of “Errors”), however all the others were attached in case they are useful for some correlation.

:small_blue_diamond: General

:small_blue_diamond: Security

:small_blue_diamond: Cache

:small_blue_diamond: Errors

:two: Site Analytics

The data that interest us are Total Requests (from both hostnames) however all the others were attached in case they are useful for some correlation.

:small_blue_diamond: hostname.gtld

:small_blue_diamond: hostname.cctld

:three: Firewall Events

Due to different subscriptions, the firewall events of hostname.gtld (Pro plan) are displayed as chart while those of hostname.cctld (Free plan) are displayed as table.

:small_blue_diamond: hostname.gtld

:small_blue_diamond: hostname.cctld


:large_orange_diamond: Logflare data

200 206 301 304 404 500
7 Feb, 7:00 PM 2525 0 2 26 0 0 2553
7 Feb, 8:00 PM 2496 1 2 37 0 0 2536
7 Feb, 9:00 PM 2964 0 6 27 0 0 2997
7 Feb, 10:00 PM 2657 3 7 27 0 0 2694
7 Feb, 11:00 PM 3019 2 5 43 0 0 3069
8 Feb, 12:00 AM 3125 0 5 50 0 0 3180
8 Feb, 1:00 AM 2805 4 1 35 0 0 2845
8 Feb, 2:00 AM 2516 2 6 20 0 0 2544
8 Feb, 3:00 AM 1751 0 1 26 0 0 1778
8 Feb, 4:00 AM 1209 0 5 8 0 0 1222
8 Feb, 5:00 AM 906 0 9 11 2 2 930
8 Feb, 6:00 AM 523 0 5 10 0 0 538
8 Feb, 7:00 AM 304 0 1 9 0 0 314
8 Feb, 8:00 AM 510 0 3 9 0 0 522
8 Feb, 9:00 AM 786 0 3 24 0 0 813
8 Feb, 10:00 AM 1227 1 3 35 0 0 1266
8 Feb, 11:00 AM 1866 0 2 19 0 0 1887
8 Feb, 12:00 PM 2315 0 1 19 0 0 2335
8 Feb, 1:00 PM 2940 1 1 27 0 0 2969
8 Feb, 2:00 PM 2900 0 3 17 0 0 2920
8 Feb, 3:00 PM 2573 2 1 18 0 0 2594
8 Feb, 4:00 PM 3108 0 1 34 0 0 3143
8 Feb, 5:00 PM 2775 0 1 26 0 0 2802
8 Feb, 6:00 PM 2857 0 6 27 0 0 2890
50657 16 80 584 2 2 51341

:large_orange_diamond: Annotations

:zero: Comparison

:small_blue_diamond: Sources

As you can see in the 🔶 Cloudflare data section, only the Primary zone has traffic. Therefore, the data in the 🔶 Comparison section refer only to hostname.gtld and were obtained from:

  • Cloudflare:
    • Site Analytics (Requests, Firewall)
    • Account Analytics (4xx errors)
  • Logflare:
    • Google Data Studio (All)

:small_blue_diamond: Timezones

All time references have been converted to UTC.

Report Timezone Start End
Cloudflare Last 24h GMT-3 2020-02-07 16:00:00 2020-02-08 15:00:00
Logflare Last 24h UTC 2020-02-07 19:00:00 2020-02-08 18:00:00

:one: Cloudflare

:small_blue_diamond: Account settings

There are only two zones assigned to this Cloudflare account:

Subscription Classification Description
hostname.gtld Pro plan Primary The main domain serving the website
hostname.cctld Free plan Secondary Additional domain redirecting to Primary

:small_blue_diamond: Analytics: Account vs. Site

These two Cloudflare tools have different report update frequency and metric shown.

Even with the same reporting period and metric time range scope, the fact that Account Analytics (Beta) displays only partial metrics while Site Analytics displays only full metrics makes it difficult to correlate the two tools through the Dashboard.

Report period Update frequency Metric time range Metric shown
Account Analytics Last 24h Almost live? 1 hour Partial
Site Analytics Last 24h 1 hour 1 hour Full
TABLE INFO
  • Report period: Total time range to which the analysed report refers
  • Update frequency.: Periodicity of data update for the selected “Report period”
  • Metric time: Time range scope that each metric “point” covers on charts
  • Metric shown:
    • Full (Report update frequency == Metric time)
    • Partial (Report update frequency > Metric time)

Unfortunately, both Site Analytics and Account Analytics documentation don’t cover the report updates neither metric interval/shown topics, so the values filled for these columns are from my observations.


:two: Logflare

:small_blue_diamond: App settings

The app has been configured to act on all requests and catch:

  • Request Method
  • Status Code
  • Connecting IP
  • Ray ID
  • Request URL
  • User Agent

:large_orange_diamond: Readings taken

:one: Cloudflare Analytics

:two: Status Codes

1 Like

Thanks for playing with Logflare!

They could be 429s which is Cloudflare rate limiting, which happens before workers get triggered. Check your firewall tab, the difference is likely there.

2 Likes

Hi @chasers,

None of the domains in this account have Rate Limiting enabled.
Could CF be preventing excessive requests even with this feature disabled?

I check the Firewall regularly and can confirm that the rate is less than 100 events/day.

By the way, It’s amazing to have apps like yours on Cloudflare!
Thank you for your awesome work!

1 Like

Yeah I’d contact support and see if they can help.

There are other Cloudflare rate limiting rules that are not listed in the firewall page. Some of them were never surfaced on the firewall page until very recently.

So, there may be something going on behind the scenes they are doing but not surfacing (except for in analytics).

1 Like

Hi @chasers,

Sorry it took so long to answer you. I really appreciate your help!

Just before you answered me I had opened a ticket with the Support Team.

Unfortunately, the last few days have been somewhat chaotic around here and I have not been able to continue the case, however now everything is back to normal.

I’ll give more details in my next post and share it with support so that we can clarify this issue - and maybe confirm we’re talking about the same thing.

Again, thank you very much for your attention!

1 Like

Dear Community and Support friends,

I completely rewrote the original publication to make the information exposed more detailed and accurate.

Thanks for everyone’s attention!

Wow great analysis! I’m glad we seem to be pretty close, but yeah unfortunately there is no visibility into those missing requests anywhere.

Note: we do use eventWaitUntil which does not guarantee the logging gets sent over to Logflare. If you’d like to run another test I can send over a custom worker that logs stuff synchronously, however that will add ~50ms or so to your response times.

2 Likes

Oh also you want to email me the Logflare url for that source? I can see if you were hitting any rate limits on the Logflare side too. You probably were not, with that volume but I can see.

edit: [email protected]

1 Like

Hi @chasers,

Thank you very much for all your attention! I’m glad you enjoyed the analysis.

Due to additional response time, can we run this custom worker for just 24 or 48 hours?

Of course, let’s eliminate that possibility.

These are the results obtained though the @chaserscustom worker that logs stuff synchronously.

Although it also covers a period of 24 hours, the new test does not correspond to the same time range as the previous one. That’s because the website loading time increased considerably, so I kept it active for the minimum time necessary.

What struck me most was that the number of 4xx errors quadrupled, exceeding the missing requests. Coincidence or not, I noticed that the Rocket Loader started having problems, resulting in several 404 errors. Since there have been no changes to the website or Cloudflare settings, I can’t suspect anything.

After completing the test, I disabled both the Logflare worker and the Rocket Loader. As a result, the Account Analytics (Beta) is showing only 259 4xx errors for the last 24 hours.


:large_orange_diamond: Round #2

All requests

Comparing Logflare to Cloudflare, there are 2,173 missing requests (~4%).

Chart data
Cloudflare [#] Logflare [#]
14 Feb, 1 AM 3984 3862
14 Feb, 2 AM 2935 2834
14 Feb, 3 AM 1699 1660
14 Feb, 4 AM 1016 960
14 Feb, 5 AM 682 650
14 Feb, 6 AM 568 510
14 Feb, 7 AM 205 194
14 Feb, 8 AM 662 617
14 Feb, 9 AM 1176 1140
14 Feb, 10 AM 1682 1637
14 Feb, 11 AM 1936 1897
14 Feb, 12 PM 2895 2826
14 Feb, 1 PM 3310 3209
14 Feb, 2 PM 3011 2932
14 Feb, 3 PM 3076 2989
14 Feb, 4 PM 2950 2863
14 Feb, 5 PM 3130 3003
14 Feb, 6 PM 2747 2664
14 Feb, 7 PM 2614 2498
14 Feb, 8 PM 2906 2847
14 Feb, 9 PM 3712 3202
14 Feb, 10 PM 3465 3357
14 Feb, 11 PM 3362 3262
15 Feb, 12 AM 3651 3588

Missing vs. Firewall

Facing the missing requests with the firewall events over time:

Chart data
Missing [#] Firewall [#]
14 Feb, 1 AM 122 4
14 Feb, 2 AM 101 28
14 Feb, 3 AM 39 7
14 Feb, 4 AM 56 34
14 Feb, 5 AM 32 13
14 Feb, 6 AM 58 22
14 Feb, 7 AM 11 7
14 Feb, 8 AM 45 23
14 Feb, 9 AM 36 5
14 Feb, 10 AM 45 8
14 Feb, 11 AM 39 7
14 Feb, 12 PM 69 9
14 Feb, 1 PM 101 11
14 Feb, 2 PM 79 16
14 Feb, 3 PM 87 8
14 Feb, 4 PM 87 4
14 Feb, 5 PM 127 8
14 Feb, 6 PM 83 11
14 Feb, 7 PM 116 8
14 Feb, 8 PM 59 3
14 Feb, 9 PM 510 202
14 Feb, 10 PM 108 21
14 Feb, 11 PM 100 5
15 Feb, 12 AM 63 1

Missing vs. 4xx Errors vs. Firewall

As the 4xx errors chart only displays the total value, this is the only possible correlation:

Chart data
Missing [#] Firewall [#] 4xx errors [#]
2173 465 3640

Logflare data

Chart data
200 206 301 304 404
14 Feb, 1 AM 3820 1 4 37 0 3862
14 Feb, 2 AM 2811 2 2 19 0 2834
14 Feb, 3 AM 1648 1 1 10 0 1660
14 Feb, 4 AM 947 0 0 13 0 960
14 Feb, 5 AM 633 0 2 15 0 650
14 Feb, 6 AM 499 0 2 9 0 510
14 Feb, 7 AM 189 0 1 4 0 194
14 Feb, 8 AM 607 0 3 7 0 617
14 Feb, 9 AM 1123 0 14 1 2 1140
14 Feb, 10 AM 1505 0 1 131 0 1637
14 Feb, 11 AM 1886 0 0 11 0 1897
14 Feb, 12 PM 2798 0 2 26 0 2826
14 Feb, 1 PM 3164 2 2 41 0 3209
14 Feb, 2 PM 2897 3 0 32 0 2932
14 Feb, 3 PM 2946 5 1 37 0 2989
14 Feb, 4 PM 2813 4 0 46 0 2863
14 Feb, 5 PM 2963 0 2 38 0 3003
14 Feb, 6 PM 2629 0 3 32 0 2664
14 Feb, 7 PM 2457 0 4 37 0 2498
14 Feb, 8 PM 2808 2 2 35 0 2847
14 Feb, 9 PM 3150 0 0 52 0 3202
14 Feb, 10 PM 3280 2 4 71 0 3357
14 Feb, 11 PM 3222 0 1 39 0 3262
15 Feb, 12 AM 3555 0 1 32 0 3588
54350 22 52 775 2 55201

:large_orange_diamond: Round #1 vs. Round #2

All requests

Missing vs. Firewall

Missings vs. 4xx Errors vs. Firewall


:large_orange_diamond: Logflare: Async vs. Sync

Super interesting. Thanks for doing this!!

The load time increase is higher than I thought it would be. Our p90 is only at about 150ms.

I wonder if support could tell you that actual 4xx status codes in the missing 4xx counts.