High packet loss on 1.1.1.1 and 1.0.0.1 today, May 23rd 2018


#1

Noticed today on my grafana ping dashboard that there is a ton of ICMP packet loss going to both of these ips.

This packet loss is not observed going towards any other of my monitored sites.

The first instance of packet loss started around 06:09:45 AM ET today and is still ongoing.

fping 1.1.1.1 1.0.0.1 -D -c 15 -s                                                                                                                                                                                                                                            130 ↵ [1527100462.583588] 1.1.1.1 : [0], 84 bytes, 4.37 ms (4.37 avg, 0% loss)
[1527100462.593077] 1.0.0.1 : [0], 84 bytes, 3.77 ms (3.77 avg, 0% loss)
[1527100464.584169] 1.1.1.1 : [2], 84 bytes, 3.89 ms (4.13 avg, 33% loss)
[1527100465.584503] 1.1.1.1 : [3], 84 bytes, 3.70 ms (3.98 avg, 25% loss)
[1527100465.594579] 1.0.0.1 : [3], 84 bytes, 3.68 ms (3.72 avg, 50% loss)
[1527100466.585515] 1.1.1.1 : [4], 84 bytes, 4.18 ms (4.03 avg, 20% loss)
[1527100466.595531] 1.0.0.1 : [4], 84 bytes, 4.11 ms (3.85 avg, 40% loss)
[1527100467.585636] 1.1.1.1 : [5], 84 bytes, 3.76 ms (3.98 avg, 16% loss)
[1527100467.595757] 1.0.0.1 : [5], 84 bytes, 3.80 ms (3.84 avg, 33% loss)
[1527100468.586688] 1.1.1.1 : [6], 84 bytes, 4.45 ms (4.05 avg, 14% loss)
[1527100468.595946] 1.0.0.1 : [6], 84 bytes, 3.62 ms (3.79 avg, 28% loss)
[1527100469.586237] 1.1.1.1 : [7], 84 bytes, 3.47 ms (3.97 avg, 12% loss)
[1527100469.596423] 1.0.0.1 : [7], 84 bytes, 3.58 ms (3.76 avg, 25% loss)
[1527100470.587201] 1.1.1.1 : [8], 84 bytes, 3.91 ms (3.96 avg, 11% loss)
[1527100470.597673] 1.0.0.1 : [8], 84 bytes, 4.30 ms (3.83 avg, 22% loss)
[1527100471.588199] 1.1.1.1 : [9], 84 bytes, 4.38 ms (4.01 avg, 10% loss)
[1527100471.598163] 1.0.0.1 : [9], 84 bytes, 4.26 ms (3.89 avg, 20% loss)
[1527100472.597815] 1.0.0.1 : [10], 84 bytes, 3.36 ms (3.83 avg, 18% loss)
[1527100473.588661] 1.1.1.1 : [11], 84 bytes, 3.79 ms (3.99 avg, 16% loss)
[1527100473.598787] 1.0.0.1 : [11], 84 bytes, 3.83 ms (3.83 avg, 16% loss)
[1527100474.588560] 1.1.1.1 : [12], 84 bytes, 3.65 ms (3.95 avg, 15% loss)
[1527100474.598436] 1.0.0.1 : [12], 84 bytes, 3.44 ms (3.79 avg, 15% loss)
[1527100475.589637] 1.1.1.1 : [13], 84 bytes, 4.20 ms (3.97 avg, 14% loss)
[1527100476.589271] 1.1.1.1 : [14], 84 bytes, 3.31 ms (3.92 avg, 13% loss)
[1527100476.599419] 1.0.0.1 : [14], 84 bytes, 3.38 ms (3.76 avg, 20% loss)

1.1.1.1 : xmt/rcv/%loss = 15/13/13%, min/avg/max = 3.31/3.92/4.45
1.0.0.1 : xmt/rcv/%loss = 15/12/20%, min/avg/max = 3.36/3.76/4.30

       2 targets
       2 alive
       0 unreachable
       0 unknown addresses

       2 timeouts (waiting for response)
      30 ICMP Echos sent
      25 ICMP Echo Replies received
       0 other ICMP received

 3.31 ms (min round trip time)
 3.84 ms (avg round trip time)
 4.45 ms (max round trip time)
       15.027 sec (elapsed real time)

#2
;; ANSWER SECTION:
id.server.              0       CH      TXT     "ewr01"

#3

Can you add a traceroute as soon as you see packetloss again? MTR with a few hundred packets would be useful :slight_smile:

Ping is good but it’s not helpful if there are any problems on the route.


#4

Sure, since it’s constant pretty easy. No loss until the route gets to cloudflare

3. b3350.nycmny-lcr-22.verizon-gni.net                 0.0%    31    5.0   5.0   2.5   7.7   1.4  
4. ???
5. 0.ae5.br1.nyc1.alter.net                            0.0%    31    4.3   4.6   3.2  22.2   3.3 
6. ae13.cr0-nyc2.ip4.gtt.net                           0.0%    31    3.4   3.5   2.8   6.9   0.8  
7. xe-8-0-0.cr0-nyc4.ip4.gtt.net                       0.0%    31    5.8   4.0   3.1   6.5   0.8  
8. cloudflare-gw.cr0-nyc4.ip4.gtt.net                 19.4%    31    4.7  12.4   3.2 168.4  33.0 
9. 1dot1dot1dot1.cloudflare-dns.com                   12.9%    31    4.6   4.0   3.4   5.0   0.4
5. 0.ae5.br1.nyc1.alter.net                                               0.0%    17    3.4   5.4   3.2  22.4   5.1  
6. ae13.cr0-nyc2.ip4.gtt.net                                              0.0%    16    2.7   3.1   2.7   4.0   0.4  
7. xe-8-2-1.cr0-nyc4.ip4.gtt.net                                          0.0%    16    4.7   4.5   4.1   5.2   0.4  
8. cloudflare-gw.cr0-nyc4.ip4.gtt.net                                    25.0%    16    3.2   7.6   3.2  39.7  10.3  
9. 1dot1dot1dot1.cloudflare-dns.com                                      25.0%    16    4.0   4.7   4.0   5.4   0.3

Ran it for longer per your request

 5. 0.ae5.br1.nyc1.alter.net                                                                0.0%   196    4.3   6.2   3.1  37.6   6.8
 6. ae13.cr0-nyc2.ip4.gtt.net                                                               0.0%   196    3.2   3.5   2.6  16.9   1.5
 7. xe-0-2-3.cr0-nyc4.ip4.gtt.net                                                           0.0%   196    3.4   4.5   3.1  24.9   1.7
    xe-11-1-2.cr0-nyc4.ip4.gtt.net
 8. cloudflare-gw.cr0-nyc4.ip4.gtt.net                                                     20.4%   196    3.6   5.4   3.1  64.8   7.5
 9. 1dot1dot1dot1.cloudflare-dns.com                                                       16.3%   196    3.5   3.8   3.3   5.1   0.4

That one is against 1.0.0.1 but they’re both similar.


#5

Now that i am at home same here up to 30% loss, constant at about 2%. Almost negligible, but…

 6. de-cix-frankfurt.as13335.net                1.5%   478    2.0   2.4   1.4  43.2   3.5
 7. 1dot1dot1dot1.cloudflare-dns.com            1.7%   478    1.4   1.4   1.2   7.4   0.3

 6. de-cix-frankfurt.as13335.net                2.1%  2266    2.0   2.2   1.5  45.8   3.9
 7. 1dot1dot1dot1.cloudflare-dns.com            2.2%  2266    1.7   1.4   1.2   7.4   0.3

@cscharff
any known issues? :thinking:


#6

Still bad this morning

8. 0.ae6.br1.nyc1.alter.net                  0.0%   416    4.7   6.7   4.1  39.4   5.6
9. ae13.cr0-nyc2.ip4.gtt.net                 0.0%   416    4.6   4.5   3.6  27.0   1.9
10. xe-0-0-0.cr0-nyc4.ip4.gtt.net            0.0%   415    5.2   5.2   4.0  40.9   2.5
    xe-8-0-0.cr0-nyc4.ip4.gtt.net
    xe-11-0-2.cr0-nyc4.ip4.gtt.net
11. cloudflare-gw.cr0-nyc4.ip4.gtt.net       14.7%   415    4.6   6.5   4.1 138.4  11.7
12. 1dot1dot1dot1.cloudflare-dns.com         21.0%   415    4.8   4.9   4.4   6.5   0.4

#7

Looks like whatever it was is fixed, thanks


#8

Cosmic radiation? :joy:

Same here.