1.1.1.1 Routing Changed Today

Zero Trust 1.1.1.1
1

Sometime in the past 3-4 hours, the routing for 1.1.1.1 changed from whatever it was, because it was working, to somewhere in Brazil. I noticed this because I foolishly had only one DNS server in configured in my router. Compared to a server I have at DreamHost, the routing is vastly different.

traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
 1  MYROUTER.MYHOMENETWORK.COM (192.168.1.1)  0.151 ms  0.104 ms  0.132 ms
 2  * * *
 3  100.41.26.132 (100.41.26.132)  6.924 ms  6.939 ms  7.006 ms
 4  0.et-10-1-5.GW15.NYC1.ALTER.NET (140.222.230.217)  13.686 ms 0.et-9-1-5.GW15.NYC1.ALTER.NET (140.222.230.215)  11.676 ms 0.et-10-1-5.GW15.NYC1.ALTER.NET (140.222.230.217)  13.684 ms
 5  embratel-gw.customer.alter.net (208.192.176.114)  21.436 ms  21.423 ms  21.510 ms
 6  ebt-B101-intl02.nyk.embratel.net.br (200.230.252.198)  131.631 ms ebt-B12211-tcore01.rjoen.embratel.net.br (200.230.220.37)  117.167 ms  125.137 ms
 7  200.244.19.54 (200.244.19.54)  126.120 ms  135.919 ms ebt-B12221-tcore01.rjoen.embratel.net.br (200.230.251.125)  139.175 ms
 8  ebt-B10-tcore01.rjoen.embratel.net.br (200.230.252.157)  124.918 ms  124.628 ms ebt-T0-0-0-iacc01.rjost.embratel.net.br (200.244.212.223)  115.942 ms
 9  200.244.19.54 (200.244.19.54)  137.414 ms 189.87.47.22 (189.87.47.22)  203.863 ms ebt-T0-0-0-iacc01.rjost.embratel.net.br (200.244.212.223)  123.446 ms
10  ebt-T0-0-0-iacc01.rjost.embratel.net.br (200.244.212.223)  125.421 ms  116.311 ms *
11  * * *
12  * * *

Note: 100.41.26.132 is Verizon, but it looks like the failure is at ALTER.NET

From my Dreamhost site:

traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
 1  ip-64-111-98-1.nodes.dream.io (64.111.98.1)  0.394 ms  0.362 ms  0.325 ms
 2  iad1-cr-1.sd.dreamhost.com (208.113.156.208)  0.323 ms iad1-cr-2.sd.dreamhost.com (208.113.156.58)  1.928 ms  1.912 ms
 3  iad1-bdr-1.sd.dreamhost.com (208.113.156.193)  0.387 ms  0.373 ms  0.366 ms
 4  ae-0-0-bdr1-iad1.dreamhost.com (208.113.156.2)  0.394 ms  0.378 ms  0.366 ms
 5  be5698.rcr51.b037327-0.iad02.atlas.cogentco.com (38.122.62.253)  0.653 ms  0.618 ms  0.626 ms
 6  be3563.ccr22.iad02.atlas.cogentco.com (154.54.25.225)  1.034 ms  1.134 ms  1.102 ms
 7  38.32.185.178 (38.32.185.178)  16.110 ms  16.114 ms  1.300 ms
 8  172.70.40.2 (172.70.40.2)  1.671 ms 173.245.63.243 (173.245.63.243)  1.070 ms  1.039 ms
 9  one.one.one.one (1.1.1.1)  0.692 ms  0.667 ms  0.641 ms

Sheer beauty.

There’s really nothing I can do about this, and 1.0.0.1 works so I am falling back to that (and other open DNS servers, but not 8.8.8.8)

If someone can get on the horn to Alter.net maybe that’s the thing to do? None of my traffic should be leaving the IAD area since that’s kinda about where I live.

(Brazil, REALLY?)

2

I noticed this as well. It’s probably only affecting Verizon customers.

For some silly reason, Verizon began accepting this BGP announcement from Embratel/Claro:

1.1.1.0/29 (2 entries, 1 announced)
        *BGP    Preference: 170/-101
                Age: 1:38:01 	Metric2: 704 
                Announcement bits (4): 0-KRT 3-RT 9-BGP_RT_Background 10-Resolve tree 4 
                AS path: 4230 27652 I  (Originator)
                Communities: 4230:21 4230:30 4230:121 
                Localpref: 100
         BGP    Preference: 170/-101
                Age: 1:38:01 	Metric2: 704 
                AS path: 4230 27652 I  (Originator)
                Communities: 4230:21 4230:30 4230:121 
                Localpref: 100

Embratel shouldn’t be announcing it, but there’s also absolutely no reason that Verizon should be accepting such a long prefix. Being the longer/more specific prefix than what Cloudflare announces (1.1.1.0/24), Verizon is preferring it and routing traffic to Brazil. Not much than can be done unless Verizon or Embratel wakes up.

3

I am having the same problem and noticed because although I have 1.0.0.1 set as an additional endpoint in the cloudflared dns over https daemon my dns was still down because 1.1.1.1 was unreachable.

The alter.net address is owned by verizon - I contacted verizon support and they gave me some runaround telling me to contact Cloudflare then they rebooted my router without my permission which of course accomplished nothing other than disconnecting me from the chat

4

Just as I finished posting above, it looks like the issue has been resolved:

1.1.1.0/24 (2 entries, 1 announced)
        *BGP    Preference: 170/-101
                Age: 4w0d 5:24:55 	Metric2: 504 
                Announcement bits (4): 0-KRT 3-RT 9-BGP_RT_Background 10-Resolve tree 4 
                AS path: 13335 I  (Originator)
                Communities: 701   13335:10386 13335:19000 13335:20050 13335:20500 13335:20530 
                Localpref: 100
         BGP    Preference: 170/-101
                Age: 4w0d 5:24:55 	Metric2: 504 
                AS path: 13335 I  (Originator)
                Communities: 701   13335:10386 13335:19000 13335:20050 13335:20500 13335:20530 
                Localpref: 100

(Also, it was never affecting 1.0.0.1 as far as I saw.)

5

…and it’s back down again. Guess they’re still asleep.

6

Looks like someone got a brain cell and things look back to normal.

traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
 1  MYROUTER.MYHOMENETWORK.COM (192.168.1.1)  0.139 ms  0.121 ms  0.112 ms
 2  * * *
 3  100.41.26.130 (100.41.26.130)  3.475 ms  3.191 ms  3.447 ms
 4  lag-15.ASBNVAEG-PPR02-CC.ALTER.NET (140.222.9.83)  5.297 ms  5.427 ms  5.383 ms
 5  204.148.11.222 (204.148.11.222)  6.466 ms  6.581 ms  6.455 ms
 6  172.70.40.4 (172.70.40.4)  6.447 ms 172.70.36.4 (172.70.36.4)  10.605 ms 172.70.172.4 (172.70.172.4)  12.537 ms
 7  one.one.one.one (1.1.1.1)  10.424 ms  8.900 ms  8.305 ms
7

Embratel leaked routes and Verizon accepted them.

For “why” this happens, see https://isbgpsafeyet.com/ - Verizon’s lack of filtering and RPKI puts users at risk.

5 Likes