Huge Performance Problem with Cloudflare

Please wait a bit, I’m still waiting for this new domains DNS propagation away from GoDaddy nameservers to Cloudflare nameservers.

2 Likes

Thank you @user2765

take the time you need to do the tests needed.

If you need anything write me i gladly will do my best to help find this problem with very bad performance for Cloudflare origin requests.

I think the DNS propagation is still causing a few issues, but give your TTFB test a try now.

1 Like

Hi @user2765

Thank you very much for your Help and testing !

I just have done a TTFB test for your Domain that points to my Origin Server with enabled Cloudflare CND Proxy and the Problem with high TTFB for Dynamic Origin Requests Disapeared.

Here is the Evidence:

How can this be ?
What exactly changed ?

I have done again a test for my domain and the TTFB Performance is same Bad.
For your Domain TTFB is Good ?

So is it ISP Related like you said ?
Its strange that @M4rt1n can not reproduce it but some can reproduce it.

I forced Cloudflare to make a connection to your origin from a POP in the same region as your Origin rather than Cloudflare reaching out to your origin from whichever POP the request first connected on.

It would also be a a good idea to re-test this https://throwaway-domain-6789.xyz/ with your other website which you were having the high TTFB issue with. You can update your origin config to add throwaway-domain-6789.xyz as an alias of that websites vhost, and then retest the TTFB.

2 Likes

Thank you very much @user2765

You helped me understand the problem and isolate it. :clap:

My understanding is in this case that Request made from Cloudflare POPs from all around the World
are not really working and having very bad Performance while only the nearest POP to Origin works best as everything else gets slowed down for what ever reasons.

By Avoiding this POP request from around the World to the Origin server you eliminated the High TTFB.

HMMM this is something very new.
It was not like this before.

Somethign Changed with Origin Requests.

This is very bad :frowning:

I will do more testing and write back.

IMHO this is something Cloudflare should fix

1 Like

I created a Cloudflare Support Ticket !

The Ticket Number is: 2460494

Hope they can solve the Problem as its affect Cloudflare Origin Requests very badly.

You’ve opened a ticket but I don’t think they will do anything about it. This is a global issue thats been going on for years intermittently from network to network, location to location. They won’t be able to fix it until they make fundamental changes to their network.

I’m happy to provide you my solution as demonstrated with https://throwaway-domain-6789.xyz/ for you to apply to your domain.

2 Likes

regarding this comment
“I forced Cloudflare to make a connection to your origin from a POP in the same region as your Origin rather than Cloudflare reaching out to your origin from whichever POP the request first connected on.”

can you please share details on how this can be done ?

I’m happy to share the details with anyone, just give me some contact info

Hi @user2765

Thank you again for your help !

Woow Crazy !
I tested the TTFB for the Origin Request the last months and never got this Problem with TTFB.

Out of Sudden the Problem appeared.
I would never expected something like this to be possible with Cloudflare.
The whole Concept of Cloudflare as CDN is broken when the requests to the Origin are not working !
This is crazy.

I make requests in workers script too and guess will need there to to make changes as any request to the Origin is now mostly broken.

Total Crazy.

Please send me your helpfull solution to => *****
If i get it to work will pay you with paypal 10 USD to buy you a coffe for helping me with the testing and providing a temprary solution till either Cloudflare fixes this problem, i change the isp hosting provider or i get full rid of any origin requests to avoid any further problems like this.

Thanks a lot and write me your paypal in the email if you want.
Best Regards.

my mail is


thx

Sent. Sent.

1 Like

Thank you @user2765

Got your eMail and your Instruction.

Could not see any paypal email written but we can do later when all is good.

Will try to understand it better.
This is quite Wizard stuff.

Btw. This bad TTFB Performance happens also on Domains where i have ARGO Smart Routing
activated and also ARGO Tiered Caching.

So despite paying for ARGO Smart Routing the Problem with very bad TTFB and Connectivity exist.
This can i confirm in my case.

Will write you back as soon i get it working.
Wish you a great Weekend.

I got a dissapointing Reply from Cloudflare Support.

Had to explain them that they need test Origin Requests and not Workers Requests that run on the POP and output a simple html …

Looks that there is a missunderstanding on your side.

What you did test using the url http://slava.mk/
was not a Request to the Origin Server
but a Worker Script that run on the POP and returns a simple html ouput.

You did not test Origin Request.
You need to use the url http://slava.mk/test.php
to make a Origin request.
PHP on the Origin server is not a problem.

What you actually is saying is to avoid any origin request using Cloudflare which in turn
makes Cloudflare as CDN total useless when Origin requests can not be maked.

Can you please Explain why there are two very different Time To First Bytes Results
when testing 1 Time the Origin Request with Enabled Cloudflare Proxy it returns catastrophic results
Measure TTFB from 35 Locations - SpeedVitals

and 1 time with disabled Cloudflare Proxy returns good Time to First Results like it should.
Measure TTFB from 35 Locations - SpeedVitals

Why are Origin Requests 1000% slowed down when turning on Cloudflare Proxy ?

Have you tried other testing tools for TTFB? Like https://tools.keycdn.com/performance ? As TTFB is dependent on origin’s geographical location - origin’s response time relative to the test server location, it stands to reason that different origins will have different TTFB from your origin’s.

Which Cloudflare plan are you using? Free, Pro, Business or Enterprise? 1. Due to varying ISP/peering costs around the world outlined by Cloudflare, the higher the Cloudflare plan you go, the better the routing and ISP/network peering and Cloudflare connectivity gets. Cloudflare refers to this in their Cloudflare plan pricing comparison table as Network Prioritization. As it may cost Cloudflare more money to serve a request from some geographic datacenters than others, financially Cloudflare can’t always route visitors to the nearest datacenter. For instance 2 geographical locations which historically showed such cases are India and Australia. On Cloudflare free plan visitors from India or Australia may be routed to Cloudflare’s Singapore or Los Angeles data centers due to lower cost.

As you moved up the Cloudflare plan tiers to more paid plans, the higher probability of getting routed closer to Cloudflare datacenters in the same region for such high cost countries. My experience with using Cloudflare for the past 11yrs on CF Free, Pro, Business and Enterprise plans, is if you want Australian visitors routed to Cloudflare Australian datacenters, you’d need at least Cloudflare Business for a higher probability of such. But for Indian visitors to be routed to Cloudflare India datacenters, you’d need a Cloudflare Enterprise plan for higher probability. I say higher probability as it still isn’t guaranteed. Though over the years, the probability has gotten better as peering arrangements have improved I suspect.

The site at cloudflare-test.judge.sh is a good guide to how peering is different for each Cloudflare plan sites. Focus on the datacenters that are reportedly served by each Cloudflare plan listed site and not the metrics which can have variance due to what individual Cloudflare sites’ settings are configured as they relate to performance i.e. if the site enabled Cloudflare Argo, Tiered Cache, Cache Reserve and any Cache Everything related Page Rules. There’s also an extended explanation at Explanation · judge2020/cloudflare-connectivity-test Wiki · GitHub.

This is what I see for http://slava.mk/test.php

for http://test.slava.mk/test.php

Noticed you configured Apache with h2c HTTP upgrade protocol - wonder if that is interfering with Cloudflare edge communications with your origin. How is PHP configured on your origin server? Which Cloudflare SSL mode do you have enabled? Full SSL or Flexible SSL?

If you’re running Cloudflare proxy in front of your origin, for optimal CF edge to origin communication, ensure the origin has HTTPS enabled with ECC 256bit ECDSA SSL certificates instead of regular RSA 2048bit SSL certificates and that HTTPS origin configuration supports TLSv1.3 so you save one round trip time (RTT) for communication https://www.cloudflare.com/learning-resources/tls-1-3/. And then ensure Cloudflare is using Full SSL mode.

2 Likes

I tested all the popular CDN networks like Cloudflare, BunnyCDN, CloudFront, Fastly, Akamai and KeyCDN, using speedvitals tool (35 test locations) and the test URL was the homepage of the respective CDN provider (must be using the best plan for their own homepage ?) to see how they compare.

And the results were comparable. The worst TTFB is always from Asia Pacific (India etc), no matter which CDN provider you use.

I am from India and can confirm positively that here in India we have many ISP providing upto 1 gbps plans, but not sure why TTFB is the worst ? Is it because ISP agreements with CDN provider are costly as mentioned by @eva2000 ?

@eva2000 can you please request CF to share some comparison data of relative cost of various ISP bandwidth across India and why TTFB latency is the worst. Bandwidth is cheap here (check Jio ISP, Airtel ISP, BSNL ISP broadband prices), not sure why the cost agreement with CDN provider is so high that you have to route it through other countries. Am I missing something?

see linked blog post in my previous reply at https://blog.cloudflare.com/bandwidth-costs-around-the-world/

Asia is 7x times the cost of Europe, South America is 17x and Oceania/Australia is 21x times.

from blog

Today, however, there are six expensive networks (HiNet, Korea Telecom, Optus, Telecom Argentina, Telefonica, Telstra ) that are more than an order of magnitude more expensive than other bandwidth providers around the globe and refuse to discuss local peering relationships. To give you a sense, these six networks represent less than 6% of the traffic but nearly 50% of our bandwidth costs.

and

If you’re a Free Cloudflare customer who cares about optimizing the best possible performance from one of these six providers then we encourage you to reach out to them and encourage them to follow a core principle of a free and open Internet and not abuse their monopoly position. We are committed to serving all our customers across every network that peers with us. To that end, help us convince these six networks to be on the right side of a free and open Internet by reaching out to your ISP.

So you’d have to ask your ISPs why they aren’t peering with Cloudflare. Right now those ISPs are prioritizing profits I suspect???

2 Likes

Not sure about other CDNs, but Cloudflare did report issues in a specific Singapore datacenter at Network Performance Issues in Singapore ~3hrs ago so that could of been one factor if your requests are routed to Singapore for Asian visitors. @M4rt1n and my tests of your site might have hit a CF Singapore datacenter that wasn’t impacted by network issues?

edit: oh wait, the tests we focused on were Europe so probably not related

1 Like