Regression with WARP on Linux?

I’m using warp-cli on Ubuntu on Include Mode purely to access some internal networks advertised with a tunnel - I’ve upgraded from 2022.2.288 to 2022.5.346 and now seem to be stuck in WarpWithDnsOverHttps mode.

Only issue is, this seems to have broken apps like Spotify and Microsoft Teams.

Teams shows errors such as: Tue Jun 21 2022 14:24:58 GMT+0100 (British Summer Time) <186871> -- error -- appOnlineSvc:Response Promise rejected during check with retryAttempt=7: Message: net::ERR_NAME_NOT_RESOLVED

I can’t quite understand why there’d be DNS resolution issues though. It’s on Include mode and shouldn’t really be touching HTTP or DNS traffic at all. If I disconnect from WARP then apps begin working pretty much instantly.

This has only become an issue since upgrading to 2022.5.346 and I couldn’t find anything in the Windows/MacOS changelogs (since Linux has none that are published?) that’d indicate what’s causing this issue.

Is anyone else experiencing the same issues?

FWIW, I’m also getting pretty constant errors when visiting sites with Hmm. We’re having trouble finding that site. (no specific error in devtools) and it takes a reload for them to actually load up since this too.

Rolled back to 2022.2.288 and I’m no-longer having these issues - and I get the mode I’d expect of just Warp.

1 Like

This is still an issue - looks like Windows has 2022.7.174.0 but Linux still has nothing newer than 2022.5.346

It goes without saying that I’m on include mode and so don’t want any of my internet-bound traffic being interfered with - but I don’t have a choice since this version of WARP forces you into WarpWithDnsOverHttps and rejects the Warp mode with Error: Invalid setting for this account type.

WARP’s DoH isn’t necessarily an issue - except when it’s breaking several apps.

Since there’s still no way that I can see to get around this, rolling back to 2022.2.288 is the only solution.

1 Like

I’m sure if you opened a ticket and posted the # here, we can escalate it to get the attention of someone on the Zero Trust team.

1 Like

Because you’re joined to a Zero Trust organization you are locked in to WarpWithDnsOverHttps mode. In prior versions of the client this was always true as well, we just hid it from you and under the covers had something like “If joined to ZT org and mode=warp then hard code mode to really = WarpWithDnsOverHttps mode”. Super confusing when debugging stuff unless you knew the magic.

Can you run warp-diag feedback and submit logs to our system? Post the ID you get mailed to you here and we can take a look.

July version of the Linux client should be out in a week or so.

Note the big change introduced with 2022.5.x was a rewrite of our DNS proxy. We now listen on the internal IPs of 127.0.2.2, 127.0.2.3, fd01:db8:1111::2, and fd01:db8:1111::3. Any chance there is a conflict with something else on the system?

2 Likes

Just curious, have you received complains from deep packet inspection not being properly disabled? We tried a new setup and applications that had cert pinning wouldn’t work despite having the TLS decryption disabled.

We began seeing this issue few weeks ago and currently it seems to be a temporal problem that comes and goes almost randomly.

1 Like

Not that I can think of - there’s no anti-virus or MDM on the devices.

The majority of websites work, but there’s two distinct symptoms:

  1. Microsoft Teams and Spotify simply refuse to load, citing network errors.
  2. It’s pretty often that the first load of a page will fail and it requires a reload.

#2 was something that I thought was just a cursed device but it seems like others have encountered the same.

On WARP 2022.2.288, there are no issues (that I can spot).

Upgrading to WARP 2022.5.346 and I can reliably reproduce #2 just by browsing through a few websites - there is no error, just a simple “unable to connect” from Firefox.

image

On the off chance it was a Firefox specific issue, it is reproducible in other browsers.

However in Chrome, I can refresh here all day long and it’ll never get anywhere - citing --> net_error = -105 (ERR_NAME_NOT_RESOLVED)

t=906 [st= 0] +HOST_RESOLVER_MANAGER_JOB  [dt=21]
               --> dns_query_types = ["A"]
               --> host = "www.reddit.com"
               --> network_isolation_key = "null null"
               --> secure_dns_mode = 1
               --> source_dependency = 1623 (SSL_CONNECT_JOB)
t=906 [st= 0]    HOST_RESOLVER_MANAGER_JOB_REQUEST_ATTACH
                 --> priority = "HIGHEST"
                 --> source_dependency = 1623 (SSL_CONNECT_JOB)
t=906 [st= 0]    HOST_RESOLVER_MANAGER_JOB_STARTED
t=906 [st= 0]   +HOST_RESOLVER_MANAGER_PROC_TASK  [dt=21]
t=906 [st= 0]      HOST_RESOLVER_MANAGER_ATTEMPT_STARTED
                   --> attempt_number = 1
t=927 [st=21]   -HOST_RESOLVER_MANAGER_PROC_TASK
                 --> net_error = -105 (ERR_NAME_NOT_RESOLVED)
                 --> os_error = -5
                 --> os_error_string = "No address associated with hostname"
t=927 [st=21]    HOST_RESOLVER_MANAGER_ATTEMPT_FINISHED
                 --> attempt_number = 1
                 --> net_error = -105 (ERR_NAME_NOT_RESOLVED)
                 --> os_error = -5
                 --> os_error_string = "No address associated with hostname"
t=927 [st=21] -HOST_RESOLVER_MANAGER_JOB
               --> net_error = -105 (ERR_NAME_NOT_RESOLVED)

Trying to launch Microsoft Teams fills the Teams logs with similar net::ERR_NAME_NOT_RESOLVED errors.

Wed Jul 27 2022 20:04:18 GMT+0100 (British Summer Time) <424462> -- info -- OnErrorOccurred - Description: UnifiedPresenceService:PublishEndpoint Method: PUT Error: net::ERR_NAME_NOT_RESOLVED 

There is slight differences in how the local DNS proxy & 1.1.1.1 return a lookup for the one of the URL’s it is having issues with, teams.microsoft.com

errorCode: -105, errorUrl: https://teams.microsoft.com, message: ERR_NAME_NOT_RESOLVED

;; QUESTION SECTION:
;teams.microsoft.com.           IN      A

;; ANSWER SECTION:
s-0005.s-msedge.net.    238     IN      A       52.113.194.132

;; ADDITIONAL SECTION:
teams.microsoft.com.    259198  IN      CNAME   teams.office.com.
teams.office.com.       298     IN      CNAME   teams-mira-afd.trafficmanager.net.
teams-mira-afd.trafficmanager.net. 58 IN CNAME  teams-office-com.s-0005.s-msedge.net.
teams-office-com.s-0005.s-msedge.net. 238 IN CNAME s-0005.s-msedge.net.

;; Query time: 16 msec
;; SERVER: 127.0.2.2#53(127.0.2.2) (UDP)
;; QUESTION SECTION:
;teams.microsoft.com.           IN      A

;; ANSWER SECTION:
teams.microsoft.com.    259188  IN      CNAME   teams.office.com.
teams.office.com.       288     IN      CNAME   teams-mira-afd.trafficmanager.net.
teams-mira-afd.trafficmanager.net. 48 IN CNAME  teams-office-com.s-0005.s-msedge.net.
teams-office-com.s-0005.s-msedge.net. 228 IN CNAME s-0005.s-msedge.net.
s-0005.s-msedge.net.    228     IN      A       52.113.194.132

;; Query time: 4 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)

Albeit both end up returning the same IP

Spotify’s logs don’t give much information beyond http_error_fail but it is reproducible going to and from 2022.2 and 2022.5

The ID I received was 507979 - thanks for the help!

1 Like

For the sake of not going too deep into the rabbit hole for something that could entirely just be an issue localized to my laptop and looking like an ■■■, I’ve tested it on a fresh install of Pop OS (just modified Ubuntu really) in a VM & it’s reproducible.

image

1 Like

Is TLS decryption disabled in Settings->Network or are you disabling it for certain applications/sites via a Do Not Inspect policy? I’ve never seen the former, please post your account id or send me a DM with it if you are in this state (or file a support ticket, dm me the support ID and I can help with routing)

The latter I’ve seen a number of times and happens for 1 of 2 reasons:

  1. The application/site has started to use a new URL/domain/ip that isnt yet in our definition for that application.
  2. There is some bug when parsing the application definition file internally

For other issues please file a support ticket letting us know

2 Likes

We have two ZT setups.
The first one has TLS decryption enabled and only has one exclusion for crowdstrike. At times there is one employee who reports that many websites become unavailable to them (despite internet being working fine).

They also reported other issues such as mailbox not loading, slack not sending messages, etc. It’s an odd issue that has been showing over the last ~2-3 weeks.
The issue isn’t limited to “external” websites, services hosted behind tunnels are also affected (the image I posted is from our GitLab instance). The employee says that disconnecting from WARP fixes the problem.

There is also this inconvenience where services such as RDP become slightly unstable during (I assume peak) hours of the day, dropping the connection every few minutes (I suspect this is unrelated to the early described issue).
image

The second issue is a new ZT setup we started on one of our customers; none of their employees could use WARP without breaking applications that relied on cert pinning.
I made sure that CF certificate was installed and that the ZT had everything related to packet inspection disabled (AV as well), but no luck.
Eventually, they gave up on enforcing WARP which isn’t optimal; I can try and convince them to give it another try.

I will collect information for both tickets and get back to you asap, any hints as to what should we provide for each ticket? a warp-diag will do? :eyes:

Thanks!

1 Like

Thank you for the detail. The issue here was our lack of handlin EDNS properly. This issue is fixed in our current builds. It won’t be fixed in the Linux build that comes out in the next few days or so but will be in the one after that.

If you see something like this in the dig results, builds prior to what I think will be 2022.8.x don’t always handle these requests properly:

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
2 Likes

Sorry to hear about these issues :frowning: warp-diag is what we need in all cases.

Note that for the certificate issues, more and more applications are relying on their own certificate store, ignoring the system one. You must also install the cert for each app via their own quirky mechanism. We’ve been documenting the ones we find here: Install the Cloudflare certificate · Cloudflare Zero Trust docs

3 Likes

At least it wasn’t just me going crazy :sweat_smile:

Thanks for taking a look into it!

Gotta love Firefox.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.