Current Network Issues

Edge servers does not have a problem. For examle if your server in turkey and your are trying to access site over cloudflare in turkey, you dont have a problem but if your server is in amsterdam or other problem locations you may get a error. Cloudflare some api servers in EU which is has a problem so you dont make a api call in web page.

My tests to OTP and IST edge servers does not have a any issue it will be reach very high speed without problem but when I try to reach AMS servers over OTP or IST it has a 500Kb/s to 10Mb/s connection.

Seems to be resolved from their end.

“This afternoon we saw an outage across some parts of our network. It was not as a result of an attack. It appears a router on our global backbone announced bad routes and caused some potions of the network to not be available. We believe we have addressed the root cause and monitoring systems for stability now.” — https://www.cloudflarestatus.com/incidents/b888fyhbygb8

3 Likes

working again now. thanks for confirming!

Dashboard is backup here too :sunglasses:

Here is my question.
Cloudflare Has a massive Network and and many customers.
Last big outage in this year is caused by lack of core server backup. If these servers is replicated in real time to other location this is not happened.
Again now I want to ask. Does the cloudflare System only one network inside ?
If the cloudflare has a many networks, and one of them has a issue system will be automatically backed up with other network.

To be honest I am failing to understand exactly the question here, but this seems to be a bad route announcement which made some IPs go to the wrong location for some regions. You can’t have different of these. The question will be why did this router announce some wrong BGP sessions?

Update from Matthew Prince, Cloudflare’s CEO.

We had an issue that impacted some portions of the @Cloudflare
network. It appears that a router in Atlanta had an error that caused bad routes across our backbone. That resulted in misrouted traffic to PoPs that connect to our backbone. We isolated the Atlanta router and shut down our backbone, routing traffic across transit providers instead. There was some congestion that caused slow performance on some links as the logging caught up. Everything is restored now and we’re looking into the root cause.


More: appears that the router in Atlanta announced bad routes (effectively a route leak). Only impacted our backbone. Not all of our PoPs are connected to our backbone, so some would not have seen an issue. Appears to have impacted about 50% of our traffic for a bit over 20 min.

https://www.cloudflarestatus.com/incidents/b888fyhbygb8

A fix has been implemented and we are monitoring the results.

If f you continue to see issues, please let support know.

1 Like

Additional info:

More: appears that the router in Atlanta announced bad routes (effectively a route leak). Only impacted our backbone. Not all of our PoPs are connected to our backbone, so some would not have seen an issue. Appears to have impacted about 50% of our traffic for a bit over 20 min.

This incident has been resolved.

https://www.cloudflarestatus.com/incidents/b888fyhbygb8

From Cloudflare Support:

With regards to the recent outage that we experienced with our DNS resolver and network, our team has identified that this issue appears to be related to our routing backbone that caused bad routes to be announced which caused some portions of the network to be unavailable.

At this time our team has implemented a fix that should resolve the issue globally and should have addressed the root cause, so this should not be occurring over the weekend;

The time stamp for Incident is between 21:11 and 22:05 UTC;

Presently a full Cloudflare service recovery should be expected; should you encounter any further issues please do let us know and we will be happy to investigate with our team;

3 Likes

Thanks for the steady updates Dom, Matteo, Matthew and everyone. We’ll be posting a blog with full details in the next few days. Sorry for the trouble all.

Yours, Otto
Head of Support, Cloudflare

8 Likes

Thanks @domjh and everyone for keeping the community updated. Full blogpost coming soon on what happened. Sorry for the trouble.

4 Likes
2 Likes

When will backbone networks be deployed in the Asia Pacific region.

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.