Cloudflare WARP and Azure Virtual Desktop (Windows365)

Zero Trust
1

Hello! We’ve been using Cloudflare on Azure Virtual Desktops for about a year with no issues. But it appears since WinDivert has been replaced with WinTun, it conflicts with something (possible suspects at the moment are: Crowdstrike Falcon, Azure Guest Agent, Microsoft RDAgent, Microsoft Geneva Agent. Microsoft Defender). That “something” causes routes to bounce first disrupting user connectivity to VM and often ends with wintun crashing. I don’t expect a solution, just posting this in case someone else is googling this and may have any ideas on how to pinpoint the culprit. No other VPN software is installed on the VMs.

  • This isn’t 100% reproducible. I’m still trying to find a pattern.
  • This happens in both “exclude” (full tunnel with Azure service endpoints excluded, so only “user” traffic is tunnelled and not RDP) and “include” (three test class A private subnets) modes.
  • The VMs have Cloudflare resolver (DoH via native Windows 11 support) configured as the default (that is – via ncpa.cpl, outside of WARP)

Log excerpt below:

2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp: Connect finished
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp: warp_start_status=Ok(())
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp_service: self.warp future resolved
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp_service: Entering main loop arm arm="tunnel_taskset_errors_fut"
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp_service: Entering main loop arm arm="network_changed"
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp_service: Network change detected new_info=IPv4: [Ethernet 2; 10.1.0.4; Ethernet; 16]; DNS servers:;   127.0.2.2:53;   127.0.2.3:53;   [fd01:db8:1111::2]:53;   [fd01:db8:1111::3]:53;   127.0.2.2:53;   127.0.2.3:53;   [fd01:db8:1111::2]:53;   [fd01:db8:1111::3]:53; old_info=IPv4: [Ethernet 2; 10.1.0.4; Ethernet; 16]; DNS servers:;   127.0.2.2:53;   127.0.2.3:53;   [fd01:db8:1111::2]:53;   [fd01:db8:1111::3]:53;   127.0.2.2:53;   127.0.2.3:53;   [fd01:db8:1111::2]:53;   [fd01:db8:1111::3]:53;
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp_service: Entering main loop arm arm="status_change"
2023-05-09T19:22:57.174Z  INFO main_loop: warp::warp_service: WARP status: Connected
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp_service::ipc_handlers: Sending IPC status update: Connected
2023-05-09T19:22:57.174Z DEBUG main_loop: warp::warp_service::ipc_handlers: Ipc Broadcast ResponseStatus: Connected
2023-05-09T19:22:59.127Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:00.143Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:00.143Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:01.142Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:01.142Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:01.373Z DEBUG warp::warp_service::network_change: Routes changed:
Added; Interface: 17; Destination: 162.159.192.0/31; Next hop: 0.0.0.0;
Added; Interface: 17; Destination: 2606:4700::2:0:0/95; Next hop: ::;
Deleted; Interface: 17; Destination: 162.159.192.1/32; Next hop: 0.0.0.0;
Added; Interface: 17; Destination: 2606:4700::1:0:0/96; Next hop: ::;
Added; Interface: 17; Destination: 2606:4700::8000:0/97; Next hop: ::;
Added; Interface: 17; Destination: 104.19.236.25/32; Next hop: 0.0.0.0;
{omitted for brevity}
Added; Interface: 17; Destination: fd01:db8:1111::3/128; Next hop: ::;
2023-05-09T19:23:02.158Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:02.158Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:03.346Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:03.346Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:04.160Z DEBUG warp_tun::win: Shutting down the wintun tunnel
2023-05-09T19:23:04.165Z DEBUG warp_tun::win: Stopping drive_read_wait_handle due to shutdown
2023-05-09T19:23:04.346Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:04.346Z DEBUG dns_proxy::errors: DnsProxy timeout target=254.169.254.169.in-addr.arpa.
2023-05-09T19:23:04.346Z  WARN trust_dns_proto::udp::udp_stream: error sending message to [2606:4700:110:86cf:ac9d:3a8a:906e:f73a]:54106 on udp_socket, dropping response: The requested address is not valid in its context. (os error 10049)
2023-05-09T19:23:04.388Z DEBUG warp::warp_service::network_change: Routes changed:
Deleted; Interface: 17; Destination: 2606:4700:0:1000::/52; Next hop: ::;
Deleted; Interface: 17; Destination: 2606:4700:0:2000::/51; Next hop: ::;
Deleted; Interface: 17; Destination: 2606:4700:0:4000::/50; Next hop: ::;
Deleted; Interface: 17; Destination: 2606:4700:0:8000::/49; Next hop: ::;
Deleted; Interface: 17; Destination: 2606:4700:1::/48; Next hop: ::;
{omitted for brevity}
Deleted; Interface: 0; Destination: ::/8; Next hop: ::;
2023-05-09T19:23:04.391Z  WARN main_loop: warp::warp: Tunnel task experienced error task_name="tun driver" err=TunDriverStopped
2023-05-09T19:23:04.391Z  WARN main_loop: warp::warp_service: Tunnel connection experienced error error=Inflight(TunDriverStopped)
2023-05-09T19:23:04.391Z DEBUG main_loop: warp::warp_service: Entering main loop arm arm="tunnel_taskset_errors_fut"
2023-05-09T19:23:04.391Z DEBUG main_loop: warp::warp::dns_recovery::windows: Reverting DNS settings old_dns=RestoreDNS { name_servers: {16: InterfaceNameServers { idx: 16, name_servers: [] }} }
2023-05-09T19:23:04.391Z DEBUG main_loop:set_dns_settings{guid="{C27FCB6D-BFD2-4A49-8CB9-71270276FD46}" nameservers="" service="Tcpip6"}: network_info::win::iphelper: Updating registry entry service="Tcpip6" nameservers=""
2023-05-09T19:23:04.391Z DEBUG main_loop:set_dns_settings{guid="{C27FCB6D-BFD2-4A49-8CB9-71270276FD46}" nameservers="" service="Tcpip"}: network_info::win::iphelper: Updating registry entry service="Tcpip" nameservers=""
2023-05-09T19:23:04.391Z DEBUG main_loop: network_info::win::iphelper: Retrieved old name servers from registry old_nameservers=[127.0.2.2, 127.0.2.3, fd01:db8:1111::2, fd01:db8:1111::3]
2023-05-09T19:23:04.391Z DEBUG main_loop: warp::warp::dns_recovery::windows: DNS settings reverted (from "successfully applied RestoreDNS { name_servers: {16: InterfaceNameServers { idx: 16, name_servers: [] }} }")
2023-05-09T19:23:04.392Z DEBUG main_loop: firewall: Firewall reset to defaults
2023-05-09T19:23:04.393Z DEBUG main_loop: wfp::session: Initializing WFP Sublayer provider=ProviderKey(79D48B13-9D76-49C8-A6CA-FC68047EC58F) name="Cloudflare WARP Firewall"
2023-05-09T19:23:04.395Z  WARN main_loop: warp::warp: Cancelled tunnel task experienced error task_name="Tunnel in/out" err=OsError(Os { code: 10022, kind: InvalidInput, message: "An invalid argument was supplied." })
2023-05-09T19:23:04.395Z DEBUG warp::warp::dns_forwarding::windows: Removed DNS IP alias
2023-05-09T19:23:04.395Z DEBUG warp::warp::dns_forwarding::windows: Removed DNS IP alias
2023-05-09T19:23:04.395Z  WARN warp::warp::dns_forwarding::windows: Failed to revert remove IP DNS address error=Os { code: 1231, kind: NetworkUnreachable, message: "The network location cannot be reached. For information about network troubleshooting, see Windows Help." } ip=fd01:db8:1111::2
2023-05-09T19:23:04.395Z  WARN warp::warp::dns_forwarding::windows: Failed to revert remove IP DNS address error=Os { code: 1231, kind: NetworkUnreachable, message: "The network location cannot be reached. For information about network troubleshooting, see Windows Help." } ip=fd01:db8:1111::3
2023-05-09T19:23:04.395Z DEBUG warp::warp::dns_forwarding::windows: Removing firewall rules for External DNS Servers ips=[fd01:db8:1111::2, fd01:db8:1111::3]
2023-05-09T19:23:04.415Z DEBUG warp::warp_service::network_change: Routes changed:
Deleted; Interface: 1; Destination: 127.0.2.2/32; Next hop: 0.0.0.0;
Deleted; Interface: 1; Destination: 127.0.2.3/32; Next hop: 0.0.0.0;
2023-05-09T19:23:04.429Z DEBUG main_loop: warp::warp_service: Reconnecting on connection error error=TunDriverStopped
2023-05-09T19:23:04.429Z DEBUG main_loop: firewall: Firewall allow private IPs
2023-05-09T19:23:04.430Z DEBUG main_loop: wfp::session: Initializing WFP Sublayer provider=ProviderKey(79D48B13-9D76-49C8-A6CA-FC68047EC58F) name="Cloudflare WARP Firewall"
2023-05-09T19:23:04.432Z  WARN main_loop: warp::warp_service: Disconnected, but reason unknown net_info=IPv4: [Ethernet 2; 10.1.0.4; Ethernet; 16]; DNS servers:;   168.63.129.16:53;
2023-05-09T19:23:04.432Z DEBUG main_loop: firewall: Firewall allow private IPs
2023-05-09T19:23:04.433Z DEBUG main_loop: wfp::session: Initializing WFP Sublayer provider=ProviderKey(79D48B13-9D76-49C8-A6CA-FC68047EC58F) name="Cloudflare WARP Firewall"
2023-05-09T19:23:04.436Z  INFO main_loop: warp::warp_service: captive_portal_fw_until: Indefinitely
2023-05-09T19:23:04.436Z DEBUG main_loop: warp::warp: Using auto fallback: true
2023-05-09T19:23:04.436Z DEBUG main_loop: warp::warp: Current Network: IPv4: [Ethernet 2; 10.1.0.4; Ethernet; 16]; DNS servers:;   168.63.129.16:53;
2023-05-09T19:23:04.436Z  INFO main_loop: warp::warp: Initiate WARP connection
2023-05-09T19:23:04.436Z DEBUG main_loop: firewall: Firewall allow tunnel
2023-05-09T19:23:04.437Z DEBUG main_loop: wfp::session: Initializing WFP Sublayer provider=ProviderKey(79D48B13-9D76-49C8-A6CA-FC68047EC58F) name="Cloudflare WARP Firewall"
2023-05-09T19:23:04.439Z DEBUG main_loop: warp::warp::happy_eyeballs: Attempting Happy Eyeballs to 162.159.192.2:2408 / [2606:4700:d0::a29f:c002]:2408
2023-05-09T19:23:04.439Z DEBUG main_loop: warp::warp::happy_eyeballs: Start racer 10.1.0.4:60090 ---> 162.159.192.2:2408
2023-05-09T19:23:04.440Z DEBUG main_loop: warp::warp::happy_eyeballs: Sent handshake initiation to 162.159.192.2:2408
2023-05-09T19:23:04.440Z DEBUG main_loop: warp::warp::happy_eyeballs: Happy eyeballs to [2606:4700:d0::a29f:c002]:2408 failed: Os { code: 10051, kind: NetworkUnreachable, message: "A socket operation was attempted to an unreachable network." }
2023-05-09T19:23:04.440Z DEBUG main_loop: warp::warp_service: Entering main loop arm arm="status_change"
2023-05-09T19:23:04.440Z  INFO main_loop: warp::warp_service: WARP status: Connecting
2023-05-09T19:23:04.440Z DEBUG main_loop: warp::warp_service::ipc_handlers: Sending IPC status update: Connecting
2023-05-09T19:23:04.440Z DEBUG main_loop: warp::warp_service::ipc_handlers: Ipc Broadcast ResponseStatus: Connecting
2023-05-09T19:23:04.448Z DEBUG main_loop: warp::warp::happy_eyeballs: Got response from 162.159.192.2:2408
2023-05-09T19:23:04.448Z DEBUG main_loop: warp::warp: Connected to 162.159.192.2:2408
2023-05-09T19:23:04.508Z  INFO main_loop: wintun: Using existing driver 0.14
2023-05-09T19:23:04.515Z  INFO main_loop: wintun: Creating adapter
2023-05-09T19:23:04.653Z DEBUG main_loop:set_dns_settings{guid="{DB484304-DB04-6AA0-A33D-7236836B364D}" nameservers="fd01:db8:1111::2,fd01:db8:1111::3" service="Tcpip6"}: network_info::win::iphelper: Updating registry entry service="Tcpip6" nameservers="fd01:db8:1111::2,fd01:db8:1111::3"
2023-05-09T19:23:04.653Z DEBUG main_loop:set_dns_settings{guid="{DB484304-DB04-6AA0-A33D-7236836B364D}" nameservers="127.0.2.2,127.0.2.3" service="Tcpip"}: network_info::win::iphelper: Updating registry entry service="Tcpip" nameservers="127.0.2.2,127.0.2.3"
2023-05-09T19:23:04.654Z DEBUG main_loop: network_info::win::iphelper: Retrieved old name servers from registry old_nameservers=[]
2023-05-09T19:23:04.657Z DEBUG main_loop:set_dns_settings{guid="{DB484304-DB04-6AA0-A33D-7236836B364D}" nameservers="fd01:db8:1111::2,fd01:db8:1111::3" service="Tcpip6"}: network_info::win::iphelper: Updating registry entry service="Tcpip6" nameservers="fd01:db8:1111::2,fd01:db8:1111::3"
2023-05-09T19:23:04.657Z DEBUG main_loop:set_dns_settings{guid="{DB484304-DB04-6AA0-A33D-7236836B364D}" nameservers="127.0.2.2,127.0.2.3" service="Tcpip"}: network_info::win::iphelper: Updating registry entry service="Tcpip" nameservers="127.0.2.2,127.0.2.3"
2023-05-09T19:23:04.658Z DEBUG main_loop: network_info::win::iphelper: Retrieved old name servers from registry old_nameservers=[127.0.2.2, 127.0.2.3, fd01:db8:1111::2, fd01:db8:1111::3]
2023-05-09T19:23:04.659Z DEBUG main_loop: firewall: Firewall allow interface iftype53_32768
2023-05-09T19:23:04.660Z DEBUG main_loop: wfp::session: Initializing WFP Sublayer provider=ProviderKey(79D48B13-9D76-49C8-A6CA-FC68047EC58F) name="Cloudflare WARP Firewall"
2023-05-09T19:23:04.664Z DEBUG main_loop: firewall: Firewall allow private IPs
2023-05-09T19:23:04.697Z DEBUG main_loop: wfp::session: Initializing WFP Sublayer provider=ProviderKey(79D48B13-9D76-49C8-A6CA-FC68047EC58F) name="Cloudflare WARP Firewall"
2023-05-09T19:23:04.767Z  INFO main_loop: warp::warp::dns_forwarding::windows: Associated unicast address and added firewall rule ip=fd01:db8:1111::2
2023-05-09T19:23:04.785Z  INFO main_loop: warp::warp::dns_forwarding::windows: Associated unicast address and added firewall rule ip=fd01:db8:1111::3
2023-05-09T19:23:04.796Z DEBUG main_loop: warp::warp::dns_recovery::windows: Applying DNS settings name_servers=[127.0.2.2, 127.0.2.3, fd01:db8:1111::2, fd01:db8:1111::3] v4_iface=Some(Ethernet 2; 10.1.0.4; Ethernet; 16) v6_iface=None
2023-05-09T19:23:04.796Z  INFO warp::warp: DnsOverWarp. Client addr is 172.16.0.2:0
2023-05-09T19:23:04.796Z  WARN trust_dns_resolver::hosts: could not parse an IP from hosts file
2023-05-09T19:23:04.796Z  INFO dns_proxy::proxy: Default fallbacks configured default_fallback_ips=[168.63.129.16:53] config=ResolverConfig { domain: Some(Name(".")), search: [], name_servers: NameServerConfigGroup([NameServerConfig { socket_addr: 168.63.129.16:53, protocol: Udp, tls_dns_name: None, trust_nx_responses: true, tls_config: None, bind_addr: None }, NameServerConfig { socket_addr: 168.63.129.16:53, protocol: Tcp, tls_dns_name: None, trust_nx_responses: true, tls_config: None, bind_addr: None }], None) } sys_options=ResolverOpts { ndots: 1, timeout: 2s, attempts: 0, rotate: true, check_names: true, edns0: true, validate: false, ip_strategy: Ipv4thenIpv6, cache_size: 32, use_hosts_file: true, positive_min_ttl: None, negative_min_ttl: None, positive_max_ttl: None, negative_max_ttl: None, num_concurrent_reqs: 8, preserve_intermediates: true, try_tcp_on_error: false, server_ordering_strategy: QueryStatistics, recursion_desired: true, authentic_data: false }
2023-05-09T19:23:04.796Z DEBUG main_loop:set_dns_settings{guid="{C27FCB6D-BFD2-4A49-8CB9-71270276FD46}" nameservers="fd01:db8:1111::2,fd01:db8:1111::3" service="Tcpip6"}: network_info::win::iphelper: Updating registry entry service="Tcpip6" nameservers="fd01:db8:1111::2,fd01:db8:1111::3"
2023-05-09T19:23:04.797Z  WARN trust_dns_resolver::hosts: could not parse an IP from hosts file
2023-05-09T19:23:04.797Z DEBUG main_loop:set_dns_settings{guid="{C27FCB6D-BFD2-4A49-8CB9-71270276FD46}" nameservers="127.0.2.2,127.0.2.3" service="Tcpip"}: network_info::win::iphelper: Updating registry entry service="Tcpip" nameservers="127.0.2.2,127.0.2.3"
2023-05-09T19:23:04.797Z DEBUG main_loop: network_info::win::iphelper: Retrieved old name servers from registry old_nameservers=[]
2023-05-09T19:23:04.823Z  WARN main_loop: trust_dns_resolver::hosts: could not parse an IP from hosts file
2023-05-09T19:23:04.920Z DEBUG warp::warp_service::network_change: Routes changed:
Deleted; Interface: 17; Destination: 162.159.193.255/32; Next hop: 0.0.0.0;
Added; Interface: 17; Destination: 192.0.1.0/24; Next hop: 0.0.0.0;
Deleted; Interface: 17; Destination: 192.0.1.255/32; Next hop: 0.0.0.0;
Added; Interface: 17; Destination: 104.19.236.128/25; Next hop: 0.0.0.0;
Deleted; Interface: 17; Destination: 104.19.236.255/32; Next hop: 0.0.0.0;
{omitted for brevity}
Added; Interface: 1; Destination: 127.0.2.3/32; Next hop: 0.0.0.0;
2023-05-09T19:23:04.925Z DEBUG main_loop: warp::warp::connectivity_check: Resolved connectivity.cloudflareclient.com to [162.159.137.65, 162.159.138.65]
2023-05-09T19:23:04.925Z DEBUG main_loop: warp::warp::connectivity_check: Resolved warp-svc. to [127.0.2.2, 127.0.2.3]
2023-05-09T19:23:04.946Z DEBUG main_loop: warp::warp::connectivity_check: fl=572f8
h=engage.cloudflareclient.com
ip=20.10.13.192
ts=1683660184.941
visit_scheme=https
uag=
colo=IAD
sliver=none
http=http/2
loc=US
tls=TLSv1.3
sni=plaintext
warp=off
gateway=off
rbi=off
kex=X25519

2023-05-09T19:23:05.004Z DEBUG main_loop: warp::warp::connectivity_check: fl=573f3
h=connectivity.cloudflareclient.com
ip=104.28.210.133
ts=1683660184.994
visit_scheme=https
uag=
colo=IAD
sliver=none
http=http/2
loc=US
tls=TLSv1.3
sni=plaintext
warp=on
gateway=off
rbi=off
kex=X25519

2023-05-09T19:23:05.004Z DEBUG main_loop: warp::warp: Trace status: Ok(TraceResult { metal_id: "573f3", timestamp: 2023-05-09 19:23:04.993999872 +00:00:00, colo: "IAD", warp: On, gateway: Off })
2023-05-09T19:23:05.004Z DEBUG main_loop: warp::warp: Connect finished
3

So far everything is pointing to Microsoft Defender. With Falcon Sensor installed on the system, Defender technically should detect the Falcon presence and switch to passive mode (pizza delivery driver mode, that is – “smell, but don’t touch”). However, it appears that as WARP loads wintun.sys, Defender attempts to insert itself to perform network protection scanning on this newly created network interface. Something was changed in this process in Defender somewhere around April this year, that breaks connectivity. When examining firewall rules, this came up:

falcon
falcon1729×607 60.8 KB

This is repeated many times, so there’s definitely a conflict of some sort. I don’t know exactly yet what LpacSenseNdr rule is trying to achieve, but judging by its name it is definitely MS Defender-related (and it’s causing issues for other apps as well). Disabling network protection seems to help, but I’m still investigating. Also, since this is happening somewhere in kernel space, adding warp-svc.exe to the exclusion list has no effect on this.

4

Spoke too soon unfortunately. Disabling Defender does not help. Even more cryptic stuff now:

Inside Tunnel
Error Tracing [connectivity.cloudflareclient.com](http://connectivity.cloudflareclient.com/) via [162.159.137.65](http://162.159.137.65/):
error sending request for url (https://connectivity.cloudflareclient.com/cdn-cgi/trace): error trying to connect: tcp connect error: An attempt was made to access a socket in a way forbidden by its access permissions. (os error 10013)
Error Tracing [connectivity.cloudflareclient.com](http://connectivity.cloudflareclient.com/) via [162.159.138.65](http://162.159.138.65/):
error sending request for url (https://connectivity.cloudflareclient.com/cdn-cgi/trace): error trying to connect: tcp connect error: An attempt was made to access a socket in a way forbidden by its access permissions. (os error 10013)
5

Ok, I give up. We are staying on the latest WinDivert version for now.

6

This has been a problem for me as well. Do you know what version is the latest that uses WinDivert?

7

I reverted to Version 2022.12.582.0, which I believe is the latest version to use WinDivert since 2023.3.381.0 mentions the new tunnel architecture. I’m still having major connectivity issues in anything except proxy mode and have been running in Include mode.

I ended up removing WARP from all of my AVD servers after having no luck reverting WARP versions and Windows updates. I’m not seeing any of the DnsProxy timeouts that you’re highlighting in the logs for 2022.12.582.0 but it does look like it’s stuck in a loop of reestablishing the tunnel over and over.

8

@amerrill yes you are correct 12.582 is the latest and yes you are correct downgrading doesn’t always help. If a system is impacted, it will remain impacted even after downgrade. And I wasn’t able to reproduce this on my test VMs no matter which WARP version and mode (include/exclude) I use, so if a system is NOT impacted it will remain not impacted. I guess the newer version doesn’t uninstall entirely from the system and leaves some misconfiguration behind.

1 Like
9

Thank you for the response. I brought up a fresh VM to verify and there is no issue prior to updating. I’ll start digging into the app and see if I can figure out what it’s leaving behind that’s causing issues, even after reverting.