Packet Loss to 1.1.1.1 from multiple networks

So I’m not hopeful Cloudflare will own any responsibility or try to correct this issue. 1.1.1.1 or it’s route from AT&T (Houston) from what I’ve found seems to be an absolute disaster, and it appears to be a problem on 1.1.1.1 Cloudflare’s end as much as I would love to cast blame on AT&T for this and their normal DNS shenanigans.

Let’s start with my home connection. AT&T

You can see. 4.16.234.182 is responding, but with significant packet loss. This appears to be inside Cloudflare 's network, but when pinging it directly, I see no issues.

So what gives?

I decided to give a few sites a try, trying to align this route. That’s when I found seattle[dot]com, which has an a record of 104.21.55.231, which is also inside Cloudflare’s network.

So what is going on? OH, and just to be clear. I have ~60 clients, various ones on different ISPs, and so far, I’ve only noticed this issue on AT&T’s network inside Houston. Not sure why that would have a factor, or if Cloudflare is doing some sort of packet shaping from AT&T but good god this issue is annoying and driving me crazy.

I have metrics on this for more than a month now with another client from a separate network entirely in the same city (Houston).

WAN 1: BIRCH/FUSION (AT&T)
WAN 2: LOGIX

Please, someone must be able to provide some insights here… this is been happening for too long.

1 Like

Also, while packet loss of ICMP traffic may not be indicative of a problem since ICMP can be deprioritized, however, when used in conjunction with routing statistics and testing against other networks which all show consistent reliable results. Why would AT&T’s network have traffic shaping & deprioritizing of ICMP traffic? Not to mention poor DNS resolution times and timeouts. That is if you’re trying to tear down the credibility of these ping results.

1 Like

Yeah, it appears that 4.16.234.182 is actually a level 3 node.

Level 3 Parent, LLC LVLT-ORG-4-8 (NET-4-0-0-0-1) 4.0.0.0 - 4.127.255.255
Level 3 Communications, Inc. LVLT-STATIC-4-16-16 (NET-4-16-0-0-1) 4.16.0.0 - 4.16.255.255

1 Like

Seeing the same issues through Level3 networks, put in a ticket to a support person and they replied with this:

First ping is to the Lumen DNS without packet loss. 1.1.1.1 belongs to Cloudfare/Mountain View. I checked our NNI to them and did not find any issues. Please engage Cloudfare on this issue or switch to the Lumen DNS which we recommend.

PING 209.244.0.3 (209.244.0.3): 56 data bytes
!!!
— 209.244.0.3 ping statistics —
1500 packets transmitted, 1500 packets received, 0% packet loss
round-trip min/avg/max/stddev = 5.446/5.899/15.409/0.837 ms

PING 1.1.1.1 (1.1.1.1): 56 data bytes
!.!!!..!..!!!..!!!.!!!..!!!.!!..!!!.!.!.!!!.!!!.!.!!!..!..!!!.!!!..!.!!!.!!!.!..!!!.!..!!!.!!!.!!!..!!!..!..!!!..!!!.!!.!..!!..!!!..!!!.!!!..!!!.!!!.!!!..!!..!!!.!..!!!.!!!.!!!..!!!.!..!!!.!!!.!!!..!!!.!!!.!!!..!!!.!!!.!!!.!!..!!!..!!!.!!.!!!..!!!.!!!.!!!.!!!.!.!..!!..!!!.!!!..!!!..!.!!!.!!!..!!.!!!.!..!!!.!!..!.!!!..!!!.!!.!!!.!.!.!!!..!.!!!..!.!!.!.!..!!.!.!!!.!!!.!!!.!!!.!.!!!.!!.!.!!!..!!!.!!!.!.!!!.!!!..!!!.!!!.!!!.!.!!!.!!!.!!!.!.!!!.!!!.!!!.!!!.!!!..!!.!!!..!!..!.!!..!!.!..!!!..!!.!!.!!!.!!!..!!.!!!..!..!!!.!!.!!!.!!!..!!!..!!!.!!!.!!..!!!.!!.!!!.!!!.!!!..!.!.!!!.!!!.!!!.!!!.!!!..!.!!!.!!.!!!.!!.!!!.!!!.!!!.!!!..!!!..!!!.!!!.!.!!!..!!.!!!.!.!!!.!!!..!.!!!.!!!.!!!..!!!..!!!.!.!!!..!!!.!!!.!!!.!!!.!!!..!..!!..!!!.!!!.!!..!!!.!!!.!!!.!..!!!.!!!..!!!..!..!!!.!!.!.!!!..!!!.!..!!.!!.!..!..!!.!.!!!.!!!.!!!.!!..!!!.!!!.!.!!!.!!!.!!!.!!..!!!.!!!.!!!.!!!.!!.!!.!..!!.!!!.!!!.!!!.!!!.!!!..!!!.!!.!..!.!!!.!!.!!!..!!..!..!!!.!.!!!.!!!.!!.!!!.!.!!!..!!!..!!!.!!!..!!!
— 1.1.1.1 ping statistics —
1500 packets transmitted, 1122 packets received, 25% packet loss
round-trip min/avg/max/stddev = 5.624/6.753/24.317/1.220 ms

1 Like

We do indeed police ICMP so one is not necessarily the reason for the other.

Can you share some more about the DNS resolution times and timeouts you are seeing?


what would you like to say for the cause of this mess?
there are no packet drops for other network. (on ethernet)

1 Like

Greetings Simon,

Is this a pattern of that ICMP Police:

Router_##ping 1.1.1.1 rep 300
Type escape sequence to abort.
Sending 300, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!.!!!.!.!!!.!!!..!.!.!.!!..!.!.!!!.!!!..!!!.!!!..!.
!.!!!..!.!!!..!!!..!.!!!..!.!.!!!.!.!.!.!!!..!!!.!!!.!
!!.!..!!!..!.!!!.!!!.!..!!!.!!!.!!!.!.!!!..!!!
!!!..!!!.!!!.!..!!!.!!!..!.!.!!..!!!.!!!.!!..!
!!!..!.!.!.!!!.!!
Success rate is 72 percent (218/300), round-trip min/avg/max = 64/71/780 ms

That test is from Venezuela.

That problem is causing failures over the dns resolution.

Hi both - given that ICMP is being dropped deliberately - those drops are not necessarily indicative of a real problem - can you share some more reproductions of the DNS failures you are seeing?

It’s not useful to look at ICMP loss alone - it would be interesting if you can correlate the loss consistently with DNS issues over a long time frame, perhaps.

1 Like

To echo or restate @simon 's point. An MTR or ping showing packet loss or drops with regards to 1.1.1.1 by itself doesn’t indicate that there are any issues with 1.1.1.1 or really any service from any provider. ICMP gets dropped by routers around the interwebs all the time for a variety of reasons.

Have problems with 1.1.1.1? Read Me First - DNS & Network / 1.1.1.1 - Cloudflare Community

None of the information in this thread demonstrates poor DNS resolution times or timeouts. If that data exists from these same locations (steps for gathering that kind of information in the thread linked above) that would be interesting to see.

Hi cscharff,

I made all the tests that appears on that link you copy ().

Let me share it with you to see if CloudFlare team see any fail:

C:\Users\username>nslookup www.marca.com 1.1.1.1
Servidor:  one.one.one.one
Address:  1.1.1.1

Respuesta no autoritativa:
Nombre:  unidadeditorial.map.fastly.net
Addresses:  199.232.197.50
          199.232.193.50
Aliases:  www.marca.com


C:\Users\username>nslookup www.marca.com 1.0.0.1
Servidor:  one.one.one.one
Address:  1.0.0.1

Respuesta no autoritativa:
Nombre:  unidadeditorial.map.fastly.net
Addresses:  199.232.193.50
          199.232.197.50
Aliases:  www.marca.com

C:\Users\username>nslookup www.marca.com 8.8.8.8
Servidor:  dns.google
Address:  8.8.8.8

Respuesta no autoritativa:
Nombre:  unidadeditorial.map.fastly.net
Addresses:  199.232.193.50
          199.232.197.50
Aliases:  www.marca.com

C:\Users\username>nslookup -class=chaos -type=txt id.server 1.1.1.1
Servidor:  one.one.one.one
Address:  1.1.1.1

Respuesta no autoritativa:
id.server       text =

        "DFW"

C:\Users\username>nslookup -class=chaos -type=txt id.server 1.0.0.1
Servidor:  one.one.one.one
Address:  1.0.0.1

Respuesta no autoritativa:
id.server       text =

        "DFW"

C:\Users\username>nslookup -type=txt whoami.cloudflare.com ns3.cloudflare.com
Servidor:  ns3.cloudflare.com
Address:  162.159.7.226

whoami.cloudflare.com   text =

        "179.6.217.63"

C:\Users\username>nslookup -vc -class=chaos -type=txt id.server 1.1.1.1
Servidor:  one.one.one.one
Address:  1.1.1.1

Respuesta no autoritativa:
id.server       text =

        "DFW"

C:\Users\username>nslookup -vc -class=chaos -type=txt id.server 1.0.0.1
Servidor:  one.one.one.one
Address:  1.0.0.1

Respuesta no autoritativa:
id.server       text =

        "DFW"

In case that there’s no issue in this tests, I can asume that the packet loss that i see is for a Policy on the cloudflare network or a policy on the ISP network.

Best Regards.

1 Like

The DNS lookups appear to succeed in your tests. Do you think there’s a DNS problem, because it doesn’t appear that there is.

As for ping failures… it’s the internet. Happens a lot for a variety of reasons (often intentionally) and is rarely in and of itself indicative of a problem.

2 Likes

Probably? I don’t really put any stock in (or pay attention to) ping tests off my own LAN because I can’t control what other operators do on the internet and there are generally other ways to test service performance or availability. Not that I don’t occasionally use ping; it’s a quick/easy test (vs. a good one) for certain things. But if it fails (consistently or intermittently) depending on what I’m trying to understand it might mean I need to switch to a better/ different test.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.