Slow TTFB after adding more firewall rules & blocking ips

[email669]

Hi,

My website TTFB has been going high randomly for the past few weeks. It’s usually more during the evenings…

But today, the TTFB has been more than double if not triple of what it normally is at this time.

Even the content download is slow… it’s normally what the TTFB is around.

Turning them off has no effect, do they need to be deleted?

This morning, I changed and added many new Firewall rules to block more traffic and wonder if this is related.

This issue seems to be globally from every edge I test.

“Root document took 270 ms” it should be 80 - 120

My questions are:

Does having more firewall rules and/or blocking make the cloudflare servers slower responding?

Can you check and see if there is something going on?

Do rules like this create a slower TTFB?

(http.user_agent contains “MJ12bot”) or (http.user_agent contains “semrush”) or (http.user_agent contains “monitorbacklinks”) or (http.user_agent contains “Uptimebot”) or (http.user_agent contains “aggregator:spinn3r”) or (http.user_agent contains “baiduspider”) or (http.user_agent contains “YandexBot”) or (http.user_agent contains “AHrefsBot”) or (http.user_agent contains “BLEXBot”) or (http.user_agent contains “Scrapy/1.5”) or (http.user_agent contains “SeznamBot”) or (http.user_agent contains “ahrefs”) or (http.user_agent contains “awario”) or (http.user_agent contains “YandexMobileBot”) + 30 more

I have checked my server and it is responding quickly and do not see any problems on my end. I can have a development subdomain and the TTFB is what it normally is.

Thank you for any help.

[sdayman]

@email669

Turning them off has no effect, do they need to be deleted?

Then that’s not the problem. Any setting that’s OFF is completely ignored.

The slow content download speed is strange. As far as I know, Cloudflare will deliver any content at full speed.

Without seeing which URL you’re testing, there’s not much we can suggest, other than the suspicion that your server has slowed down for some reason.

[email669]

Hi,

please look at the deleted post for more information.

It almost appears as if Cloudflare is no longer keeping the session alive and doing a new connect for every request or there are rules or things slowing the cloudflare response.

I have a TTFB from the source link on 2nd request in 60 MS, through cloudflare it is 200 - 300 ms.

Second response through Cloudflare:

Second response through origin:

[sdayman]

I looked at your deleted stats. So your concern is an additional 66ms for TTFB? That 66ms most likely comes from the extra hop it takes for Cloudflare to grab your non-cached HTML content from the origin server. I most definitely wouldn’t consider an additional 66ms a “slow” TTFB. Especially when the rest of the content you’re serving through Cloudflare is super fast. If you want to speed up the site, slim it down and drop all the ads.

[email669]

No, it’s the additional 150+ MS for the main page load that just started today. It get’s worse towards the west coast.

I am not concerned with “page speed” but how fast the cloudflare server responds to the host and how fast the html is rendered to them.

If my origin server is rendering the content to me in 60 MS, why is Cloudflare taking over 200 MS to render the same content?

Where is the 200 MS coming from when i’m connecting to a server in the same state as the cloudflare server? The page renders in 60 ms.

I understand this page is uncached and it must request from the origin, but it has always been 90 to 120 MS response until today and everywhere is showing way slower response.

My server load is OK and here is my first load request from the server.

Origin First Response

Second Response through Cloudflare

Notice the same response time as through cloudflare the second request? + about 40 ms for cloudflare to connect in state and retrieve the page.

Cloudflare used to stay connected and retrieve the second response quicker.

[email669]

Please see a speed test here for the FIRST request and it is over 800 MS!?!? 500MS?

May 12th Tests

https://gtmetrix.com/reports/www.infinitesweeps.com/KIR3pq7g
https://gtmetrix.com/reports/www.infinitesweeps.com/9UhGdZUk

2.4 seconds?

https://gtmetrix.com/reports/www.infinitesweeps.com/ik2fhd8i

Please see the previous waterfalls here:

May 1st Test

https://gtmetrix.com/reports/www.infinitesweeps.com/TKBo6f12
https://gtmetrix.com/reports/www.infinitesweeps.com/pNkqQSEF

Something is wrong. I did block a bunch of bots today in Cloudflare.

[email669]

It’s starting and stopping. This is some type of attack it appears. It was slowly getting better and then goes back up to 2+ seconds down to regular and starts over.

Is there something I should do? I don’t see any requests or log events. My server cpu usage is 5% and there are no abnormal logs to anything on my end.

If this was a non hit image, it would mean that it would take 2+ seconds to render a 12 KB size image. 1 out of 3 or 4 tests shows 2.5 seconds+

It was always steady at under 200 MS tops around the entire USA. Now it’s 300MS to 2.6 seconds. It went back to normal a few times.

[M4rt1n]

I reported the same (or similar) behaviour some days ago.

Slow Website if routed through CF

I do not use CloudFlare “Cache Everything” because of dynamic content. But I noticed at my Website which uses “FRA” as CloudFlare-Ray that my TTFB is very very high. When I call a site which is getting routed through CloudFlare-Ray “FRA” I get a TTFB of about 500ms-600ms [2020.05.02 TTFB CloudFlare] The response Header looks like this: [2020.05.02 TTFB CloudFlare Header] When I bypass CloudFlare (with curl) my TTFB is at 40ms. Also when I call other sites which are static (like this one…

I dont know where this comes from, but as the files are cached by Cloudflare, this can not be related to my Server. Everything after the server should be CloudFlares “problem” and the at least should investigate where these comes from, as just saying “its your ISPs” fault is very very cheap…

@sdayman

The slow content download speed is strange. As far as I know, Cloudflare will deliver any content at full speed.

I atm also suffer from this. 2kb takes 500ms to download, TTFB is very bad also. But as I said it depends very hard from every minute to minute.

In germany we have just 3-4 ISP, so at least 30% of german users are actually atm getting fuucking bad Cloudflare experiance and performance.
Thats sad.

BTW: my normal TTFB is about 30ms. And I always had about 30ms with Cloudflare, untill some days when I reported this. This issue is still now here. Sometimes it is, sometimes not…

[email669]

I am seeing the same thing and even high TTFB responses even from image requests. 200 MS.

I was messing with Cloudflare Workers a week or so ago and noticed the same behavior, so I chose not to use them.

I have completely removed the workers, but it’s almost like there is a worker waiting until all content is there before responding. Not sending until all information is received.

I’ve opened a ticket with Cloudflare, and they are confirming my server is responding slower - but it’s not. My CPU usage is 5% and it can easily go up to 50% before start seeing effects in response times.

I hope they come back with more information, here is a graph they have provided - I do not see any lag on my end and receive 60 MS response times.

Here is my CPU usage on a dedicated at the time it’s still happening:

[cuded2]

Checked on Google Pagespeed:

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwww.infinitesweeps.com%2Fsweepstake%2F197787-Cycle-Gear-Irodetoday-Motorcycle-Sweepstakes.html&hl=en

It seems TTFB of this page is not an issue, but FID. Because the FID is over 300ms, your page is labeled as “Slow” even TTFB is further improved. Check out Google Console on the Speed performance for this site.

[email669]

Hi cuded2,

The FID score you are looking at is based on the 30 day average. If you look now, you will see a much lower “Lab Data” score since I have implemented changes to the server side of the scripting a few weeks ago that will decrease the FID. (javascript)

The reason I am posting is because of the random TTFB delays on requests. This has nothing to do with the client side of things. This morning, it does seem better, but not as “stable” as it previously was.

I do think the higher FID from real world experience is from these bad bots I have been trying to block for years… Unless you see something else causing a high FID that I am missing…

[cuded2]

@email669

Yes, it’s average result of the last 30 days on a group of similar pages on your site. In other words, other pages may have impact on the average. If you just made optimization, the average should go down day by day. Today is 373ms on my end. If there’s no change, further optimization is necessary.

[email669]

Hi,

It should right?

I made these optimizations about a month ago and since then, the numbers are going the wrong way.

After doing these optimizations I have been having trouble with the Cloudflare server responding with random times that I think was affecting the FID. I believe due to competitor bot attacks.

When I run the Google Search Console speed test, it shows an average FID of 310 MS, which I wonder if is from the random and sometimes super high response times.

Things seem much better the last day or two, so I will hope for the best.

If you see anything on my FID that can be optimized, please let me know. Every test I run shows the highest delay of up to 150 MS with 6CPU slowdown.

https://developers.google.com/speed/pagespeed/insights/?url=https%3A%2F%2Fwww.infinitesweeps.com%2Fsweepstake%2F197787-Cycle-Gear-Irodetoday-Motorcycle-Sweepstakes.html&hl=en&tab=mobile

Thank you!!!

[eva2000]

FID = first input delay https://almanac.httparchive.org/en/2019/performance#first-input-delay

This metric represents the time from a user’s first interaction with a page’s UI until the time the browser’s main thread is ready to process the event. Note that this doesn’t include the time applications spend actually handling the input. At worst, slow FID results in a page that appears unresponsive and a frustrating user experience.

If you read the Web Almanac 2019 report https://almanac.httparchive.org/en/2019/performance#first-input-delay , you’ll see that like FCP - first contentful paint, FID is dependent on browser/device cpu speed, geographic location of visitors and speed of visitor’s ISP connection. Hence, the difference you see in Google PageSpeed Insights field and origin metrics for mobile vs desktop results. As mobile is tested on Nexus 4 mobile device with 3G mobile speed profile emulation for the lab data test portion.

So you could have 2 sites with same theme with different FID and FCP results due to the 2 sites having different visitor traffic profiles for geographic location, browser/device used and ISP connection speed.

Google Webmaster Console’s Speed report is based on Chrome User Experience data (CRuX) which is real world Chrome visitor data for your sites’ pages - what Google calls field data as opposed to lab data of which both field and lab data make up the components of Google PageSpeed Insights v5 https://developers.google.com/speed/pagespeed/insights/ results.

[email669]

@eva2000

So you could have 2 sites with same theme with different FID and FCP results due to the 2 sites having different visitor traffic profiles for geographic location, browser/device used and ISP connection speed.

"By breaking FID down by device, it becomes clear that there are two very different stories. Desktop users enjoy fast FID almost all the time. Sure, there are some websites that throw out a slow experience now and then, but the results are predominantly fast. Mobile users, on the other hand, have what seem to be one of two experiences: pretty fast (but not quite as often as desktop) and almost never fast. The latter is experienced by users on only the tail 10% of websites, but this is still a substantial difference.

When we apply the PSI labeling to desktop and phone experiences, the distinction becomes crystal clear. 82% of websites’ FID experienced by desktop users are fast compared to 5% slow. For mobile experiences, 26% of websites are fast while 22% are slow. Form factor plays a major role in the performance of interactivity metrics like FID.

On its face, FID seems like it would be driven primarily by CPU speed. It’d be reasonable to assume that the slower the device itself is, the higher the likelihood that it will be busy when the user attempts to interact with a web page, right?

The ECT results above seem to suggest that there is a correlation between connection speed and FID performance. As users’ effective connection speed decreases, the percent of websites on which they experience fast FID also decreases: 41% of websites visited by users with a 4G ECT have fast FID, 22% with 3G, 19% with 2G, and 15% with slow 2G."

" A site is slow if 5% of its FID experiences are slow. All other experiences are moderate .", could be the reason why I had such a huge spike from “fast” to “slow” in the real world experiences.

Hi,

Everything you said is what i’m talking about…

FID has to do with user speed, so even if 1 of the images on the page has a 2.0 second load time, it will affect everything else about the page and increase the FID if it’s on the main thread.

If the main html page takes 2.0 seconds to load, everything is now 1.9 seconds behind the usual loading.

“I scrolled down and the lazy loaded image took 2 seconds to load” = 2 second FID for this users interaction.

Anyway, thank you for replying and I hope for the best.

[cuded2]

@email669

FID has nothing to do with the server’s performance, but your on-page optimization. In your case, the problem may lie in the javascript for adsense loading. As your adsense code is loaded by the script, it seems the adsense code is loaded twice. See the screenshot.

So you can try to load adsense normally. In addition, there’s no need to preload adsense code. Just load it asynchronously. This will have good impact on Largest Contentful Paint performance (Google newly launched metric).

I also noticed you used resource hints in the body rather than header.