Intermittent connectivity loss to 1.1.1.1 and 1.0.0.1 from Chicago


#1

Multiple times a day I lose connectivity to 1.1.1.1 for several minutes at a time. All else seems ok, I can access 8.8.8.8 just fine. My ISP (Everywhere Wireless) says things look good from their end but are eager to help troubleshoot further as needed. I can trace route all the way up to Cloudflare’s IP 206.223.119.180 (prev hop
is RCN (another ISP in Chicago) 207.172.19.161) and then it times out. Below is a screenshot of trace route showing during and after one of the incidents (error and success).

Imgur

Everything that follows here is while it is working. I will try to access those diagnostics again (now that I’m aware of them) the next time it goes down and will update.

Tracing route to 1.1.1.1 over a maximum of 15 hops

1 <1 ms <1 ms <1 ms 10.0.0.1
2 1 ms <1 ms <1 ms 10.14.180.1
3 <1 ms <1 ms <1 ms BLANKEDOUT
4 1 ms <1 ms <1 ms 104.153.229.85
5 1 ms 1 ms 1 ms 216.80.0.165
6 2 ms 1 ms 2 ms 207.172.19.158
7 2 ms 1 ms 1 ms 207.172.19.161
8 2 ms 5 ms 2 ms 206.223.119.180
9 1 ms 1 ms 1 ms 1.1.1.1

Trace complete.

Diagnostic: https://cloudflare-dns.com/help/#eyJpc0NmIjoiWWVzIiwiaXNEb3QiOiJObyIsImlzRG9oIjoiWWVzIiwicmVzb2x2ZXJJcC0xLjEuMS4xIjoiWWVzIiwicmVzb2x2ZXJJcC0xLjAuMC4xIjoiWWVzIiwicmVzb2x2ZXJJcC0yNjA2OjQ3MDA6NDcwMDo6MTExMSI6Ik5vIiwicmVzb2x2ZXJJcC0yNjA2OjQ3MDA6NDcwMDo6MTAwMSI6Ik5vIiwiZGF0YWNlbnRlckxvY2F0aW9uIjoiT1JEIiwiaXNwTmFtZSI6IkNsb3VkZmxhcmUiLCJpc3BBc24iOiIxMzMzNSJ9

HTTPS Diagnostic: {“Status”: 0,“TC”: false,“RD”: true, “RA”: true, “AD”: true,“CD”: false,“Question”:[{“name”: “cloudflare.com.”, “type”: 1}],“Answer”:[{“name”: “cloudflare.com.”, “type”: 1, “TTL”: 23, “data”: “198.41.214.162”},{“name”: “cloudflare.com.”, “type”: 1, “TTL”: 23, “data”: “198.41.215.162”}]}


#2

The 206 address is actually not a Cloudflare address but presumably belongs to the datacentre where Cloudflare is hosting in this case, which appears to be the actual Chicago DC.

To me this looks like some intermittent connectivity issue right in the datacentre, but a definite answer would require a deeper analysis. Thats definitely a case for support though.

@cloonan @cscharff @ryan


#3

Thanks for the insights. Please let me know if there is any further information I can collect.

I monitored through this afternoon and lost connectivity again from roughly 2:08pm-2:13pm CST, and again at roughly 3:08pm-3:13pm CST. I, however, did not lose connectivity at 4:08pm for whatever reason.

FWIW too, I wasn’t able to access either of the debug pages by IP, so I could not get any further logs during the outages.


#4

What you could additionally check is whether it might possibly already stop before 206 (hop 8) when it stops. Simply run a few concurrent traceroutes during an outage, all slightly shifted timewise. Even though unlikely, it could be packets got dropped already at e.g. hop 5 in your first traceroute.

You could also run http://winmtr.net/


#5

Everything looks good up until 206 still sadly. Though I haven’t had any frequent daytime issues like I had for the prior two weeks, I’m still having issues around 10 or 11p each night.

Is there anything else I can do to make sure this gets looked at on Cloudflare’s end?


#6

If it is happening after 206 it likely would point towards an issue in Cloudflare’s datacentre.

Have you already opened a support ticket? Maybe also @ryan and @cloonan can have a look.


#7

Yes, please share the ticket number.


#8

@sandro @cloonan when I opened a ticket, they told me to come here. The ticket number was #1619878 for what it’s worth though.

Thanks!


#9

Support told you to ask in the forum?! :face_with_raised_eyebrow:

Anyhow, the ticket number should be enough for @cloonan :slight_smile:


#10

Hi @gotdibbs, thank you, I’ve added myself to the ticket and reopened it.