MSP seeing all of our Michigan Comcast customers experiencing 25 to 40% packet loss to CloudFlare DNS 1.1.1.1 and 1.0.0.1

Hi everyone, we’ve been big fans of the cloudflare DNS resolvers and use it for all of our I.T. customers, but while diagnosing several internet issues the past few days we noticed that across the board every single one of our customers using Comcast as their ISP (Home & Business) are getting 25% to 40% packet loss to IP 1.1.1.1 and 1.0.0.1. As far as we know, it does appear isolated to Cloudflares dns resolver IPs and I’m unaware if this is affecting Comcast users outside metro Detroit. Traceroute shows the possible problem child being 173.167.56.42 .

I’ve tried reporting this to Comcast technical support and escalating it but I’m sure anyone that has dealt with that knows I’m spinning my wheels. Does anyone have any suggestions?

I’m sure you realize that also belongs to Comcast.

At this end, @mvavrusa does a lot of the 1.1.1.1 work, and @cscharff has been known to jump in from time to time.

You may have seen this before, but pasting in as many test results from that as possible would help.

Yea figured it was a Comcast IP and confirmed with a quick arin search. Was just hoping someone here would have a possible different angle to get Comcast to fix this.
That link is super cool. I’ll run it on a few locations and post results. Thanks!

Here’s 6 different results from that tool from 6 different service locations in about a 50 mile radius. 5 Comcast Business & 1 Comcast Home

A few traceroutes to 1.1.1.1 and 1.0.0.1 would help confirm the bottleneck.

sure thing!
image


image
image

interesting there’s no spikes in the CF DNS response time - just hard drops.

Edit: connected to a client over in the west coast over VPN and saw zero drops.

As this appears to be outside of CF and our control, i’ll be moving our clients over to Level3 and Google until there’s some progress made. If a ticket or tech needs additional information for progress, I’ll be glad to assist.

This is certainly rough.
The problem pretty much confirmed, now it’s just a question of when/if CF/Comcast will act on it.

Open to helping, just let me know.

So this is just dead in the water I guess?

Open a ticket and include a link to this thread.

To contact Cloudflare Customer Support, login & go to https://dash.cloudflare.com/?account=support and select get more help. If you receive an automatic response that does not help you, please reply and indicate you need more help.

Thank you sir!

Im now stuck on their ticket submissions form lol. Says its running diagnostics on my domain and if I still have issues to submit a ticket… But it wont let me…

Drop them an email at [email protected].

1 Like

Can we just set it as sticky one day, give it to one of interim to blog.

ICMP traceroutes, and observed packet loss (or timeouts), are not indicative of a problem, routers will drop/block/depriotize ICMP traffic. Do you really believe that CF would not be aware of 25%+ packet loss to their edge servers from perhaps the biggest eyeball network outside of China?

Yikes

This (and the thoudsands other threads OMG look at this traceroute all those *) are why [email protected] is almost always blackholed.

You obviously didn’t read the post in its entirety, or you’re insinuating that CF is dropping ICMP traffic on purpose. Also I don’t believe that Michigan Comcast customers are the biggest eyeball network outside of China, but you apparently know more than I do plus it does sound pretty cool. Thank you for your contribution to this post! It’s helped a great deal!

I’ve contacted comcast regarding this issue because I am also experiencing it from the Chicago area. This was tier 2 enterprise support and they recognized the issue and identified an existing ticket regarding a link that is over saturated. The resolution involves very expensive upgrades to the equipment on that link, but they are actively working on it. Apparently this requires executive approval due to the costs involved. I’m guessing it will be a few weeks before it is resolved. Very frustrating, with only a few million people affected apparently it’s not a huge priority.

1 Like

Thank you for that update!