WARP Split-tunnel not working as expected

Hi everyone!

I’m trying to reach my home network through Cloudflare WARP (for Teams) but I keep running into issues. I believe that everything has been configured correctly.

Some details about my setup:

  • Home network is running in the range 192.168.2.0/24
  • I’m running cloudflared on a virtual machine inside the main network.
    • Version: 2021.7.0 (built 2021-07-12-1109 UTC)
    • I’ve used this in the past to expose my Home Assistant instance to the web. This is working well, meaning cloudflared does have access to the network (no firewall rules yet)
    • Now I want to reach the entire network (without having configure ingress rules for every service)
  • Running Cloudflare WARP client on macOS v1.5.294.0 (20210701.2)

Here’s my config file for cloudflared:

tunnel: tunnel-home-assistant 
credentials-file: /root/.cloudflared/9fa2b695-0306-4b27-abca-ff689ab5518d.json

warp-routing:
  enabled: true

logfile: /var/log/cloudflared
loglevel: trace
transport-loglevel: trace
no-tls-verify: true

ingress:
  - hostname: xxxxxxxxx
    service: http://192.168.2.95:8123
  - service: http_status:404

I enabled warp-routing, and no-tls-verify because some HTTPS services on my network use self-signed certificates. The ingress rule works as expected.

Problem: The WARP split-tunnel doesn’t work as expected. I can’t reach any service running within my home network when connected through WARP. When I try to open a HTTPS endpoint, I get this error:

(Chrome just keeps on spinning)

When trying to reach a regular HTTP endpoint, I get this:

Which seems to indicate that Cloudflare is at least connecting to something in my home network?

I checked the Gateway logs (Logs > Gateway) and traffic to my local network is allowed:

Gateway with WARP is enabled on the Mac application:

7397E0A3-4AC4-45FB-95F2-D6FAD549D90B

I’m also logged into my Teams account:

And my home network (192.168.2.0/24) is not in the “Excluded IPs” section. I removed it using the Teams Dashboard > Settings > Network > Split Tunnels, and that change has been reflected in the macOS client:

I have read through the documentation pages, and I believe that everything is configured correctly. I’m hoping that someone can point me in the right direction to solve this problem.

Many thanks!

Hello,

Based on what you showed, it looks like traffic is getting from your MacOS WARP client to Cloudflare edge, because you showed a screenshot of Gateway’s Log where 1 of those requests is visible.

So now the question is whether it is making it to your cloudflared or not.
You have already enabled a high verbosity logging level on cloudflared. Do you see anything there when you try to access from your MacOS WARP client by IP?

Also, can you make sure you can access that IP:port from the VM/machine where cloudflared is running?

Cheers,
– Nuno

Hi Nuno,

Thanks for your reply! No, the log file only shows that cloudflared occasionally lost connection over the last few months and automatically reconnected. Nothing alarming.

To verify, I cleared the log file, restarted cloudflared and tried access the local network again. This is the resulting log file, no messages about traffic hitting it:

{"level":"info","tunnelID":"9fa2b695-0306-4b27-abca-ff689ab5518d","time":"2021-07-26T17:00:58Z","message":"Starting tunnel"}
{"level":"info","time":"2021-07-26T17:00:58Z","message":"Version 2021.7.0"}
{"level":"info","time":"2021-07-26T17:00:58Z","message":"GOOS: linux, GOVersion: devel +11087322f8 Fri Nov 13 03:04:52 2020 +0100, GoArch: amd64"}
{"level":"info","time":"2021-07-26T17:00:58Z","message":"Settings: map[config:/root/.cloudflared/config.yml cred-file:/root/.cloudflared/9fa2b695-0306-4b27-abca-ff689ab5518d.json credentials-file:/root/.cloudflared/9fa2b695-0306-4b27-abca-ff689ab5518d.json logfile:/var/log/cloudflared.log no-tls-verify:true proto-loglevel:trace transport-loglevel:trace]"}
{"level":"info","time":"2021-07-26T17:00:58Z","message":"Generated Connector ID: 5eb5621b-79da-4796-bde4-c417fe4bf7fd"}
{"level":"info","time":"2021-07-26T17:00:58Z","message":"Warp-routing is enabled"}
{"level":"info","time":"2021-07-26T17:00:58Z","message":"Initial protocol http2"}
{"level":"info","autoupdateFreq":86400000,"time":"2021-07-26T17:00:58Z","message":"Autoupdate frequency is set"}
{"level":"info","time":"2021-07-26T17:00:58Z","message":"Starting metrics server on 127.0.0.1:40483/metrics"}
{"level":"info","connIndex":0,"location":"AMS","time":"2021-07-26T17:00:59Z","message":"Connection d6a6cbfa-81f0-41d5-8b7a-725b99eec529 registered"}
{"level":"info","connIndex":1,"location":"LHR","time":"2021-07-26T17:00:59Z","message":"Connection 1aecb1c8-89e6-42f3-a9b6-d78e19fa5dc5 registered"}
{"level":"info","connIndex":2,"location":"AMS","time":"2021-07-26T17:01:00Z","message":"Connection d085b899-da95-4291-a08d-20d4a551fcf7 registered"}
{"level":"info","connIndex":3,"location":"LHR","time":"2021-07-26T17:01:01Z","message":"Connection 9c3c61b5-4704-4c71-949b-b5f630f448a7 registered"}

I tried this with my Mac connected to the local network and through a 4G hotspot.

Mac -> 4G -> WARP -> Split tunnel
Mac -> Local WiFi -> WARP -> Split tunnel

I can confirm that the VM can access the entire network. The ingress rule that maps one HTTPS service to a subdomain does work fine (192.168.2.95). Only the split-tunnel is not working.

Can you switch the loglevel to debug?
I would expect the log to have a lot more verbosity than what you showed.

Not seeing anything specific in debug mode either. Seems like traffic is not getting to cloudflared?

{"level":"info","tunnelID":"9fa2b695-0306-4b27-abca-ff689ab5518d","time":"2021-07-26T17:38:16Z","message":"Starting tunnel"}
{"level":"info","time":"2021-07-26T17:38:16Z","message":"Version 2021.7.0"}
{"level":"info","time":"2021-07-26T17:38:16Z","message":"GOOS: linux, GOVersion: devel +11087322f8 Fri Nov 13 03:04:52 2020 +0100, GoArch: amd64"}
{"level":"info","time":"2021-07-26T17:38:16Z","message":"Settings: map[config:/root/.cloudflared/config.yml cred-file:/root/.cloudflared/9fa2b695-0306-4b27-abca-ff689ab5518d.json credentials-file:/root/.cloudflared/9fa2b695-0306-4b27-abca-ff689ab5518d.json logfile:/var/log/cloudflared.log loglevel:debug no-tls-verify:true proto-loglevel:debug transport-loglevel:debug]"}
{"level":"info","time":"2021-07-26T17:38:16Z","message":"Generated Connector ID: b17b5497-ea46-497c-b4d1-6db7788ce688"}
{"level":"info","time":"2021-07-26T17:38:16Z","message":"Warp-routing is enabled"}
{"level":"info","time":"2021-07-26T17:38:16Z","message":"Initial protocol http2"}
{"level":"info","autoupdateFreq":86400000,"time":"2021-07-26T17:38:16Z","message":"Autoupdate frequency is set"}
{"level":"info","time":"2021-07-26T17:38:16Z","message":"Starting metrics server on 127.0.0.1:37559/metrics"}
{"level":"debug","connIndex":0,"address":"198.41.200.53:7844","time":"2021-07-26T17:38:16Z","message":"edgediscovery - GetAddr: Giving connection its new address"}
{"level":"debug","connIndex":0,"time":"2021-07-26T17:38:16Z","message":"Connecting via http2"}
{"level":"debug","time":"2021-07-26T17:38:16Z","message":"rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))"}
{"level":"debug","time":"2021-07-26T17:38:16Z","message":"rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))"}
{"level":"debug","time":"2021-07-26T17:38:16Z","message":"rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))"}
{"level":"debug","time":"2021-07-26T17:38:16Z","message":"rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))"}
{"level":"debug","time":"2021-07-26T17:38:17Z","message":"rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))"}
{"level":"info","connIndex":0,"location":"AMS","time":"2021-07-26T17:38:17Z","message":"Connection 4a1b05e2-7743-444d-9e8a-c392a507497a registered"}
{"level":"debug","time":"2021-07-26T17:38:17Z","message":"rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))"}
{"level":"debug","connIndex":1,"address":"198.41.192.107:7844","time":"2021-07-26T17:38:17Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
{"level":"debug","connIndex":1,"time":"2021-07-26T17:38:17Z","message":"Connecting via http2"}
{"level":"debug","time":"2021-07-26T17:38:17Z","message":"rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))"}
{"level":"debug","time":"2021-07-26T17:38:17Z","message":"rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))"}
{"level":"debug","time":"2021-07-26T17:38:17Z","message":"rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))"}
{"level":"debug","time":"2021-07-26T17:38:17Z","message":"rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))"}
{"level":"info","connIndex":1,"location":"LHR","time":"2021-07-26T17:38:18Z","message":"Connection 02dfc740-7964-4a8c-846d-255d089db296 registered"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))"}
{"level":"debug","connIndex":2,"address":"198.41.200.13:7844","time":"2021-07-26T17:38:18Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
{"level":"debug","connIndex":2,"time":"2021-07-26T17:38:18Z","message":"Connecting via http2"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))"}
{"level":"info","connIndex":2,"location":"AMS","time":"2021-07-26T17:38:18Z","message":"Connection d07cd349-56d2-448a-810c-63ed18063c2e registered"}
{"level":"debug","time":"2021-07-26T17:38:18Z","message":"rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))"}
{"level":"debug","connIndex":3,"address":"198.41.192.167:7844","time":"2021-07-26T17:38:19Z","message":"edgediscovery - GetDifferentAddr: Giving connection its new address"}
{"level":"debug","connIndex":3,"time":"2021-07-26T17:38:19Z","message":"Connecting via http2"}
{"level":"debug","time":"2021-07-26T17:38:19Z","message":"rpcconnect: tx (bootstrap = (questionId = 0, deprecatedObjectId = <opaque pointer>))"}
{"level":"debug","time":"2021-07-26T17:38:19Z","message":"rpcconnect: tx (call = (questionId = 1, target = (promisedAnswer = (questionId = 0, transform = [])), interfaceId = 17804583019846587543, methodId = 0, allowThirdPartyTailCall = false, params = (content = <opaque pointer>, capTable = []), sendResultsTo = (caller = void)))"}
{"level":"debug","time":"2021-07-26T17:38:19Z","message":"rpcconnect: rx (return = (answerId = 0, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [(senderHosted = 0)])))"}
{"level":"debug","time":"2021-07-26T17:38:19Z","message":"rpcconnect: tx (finish = (questionId = 0, releaseResultCaps = false))"}
{"level":"debug","time":"2021-07-26T17:38:19Z","message":"rpcconnect: rx (return = (answerId = 1, releaseParamCaps = false, results = (content = <opaque pointer>, capTable = [])))"}
{"level":"info","connIndex":3,"location":"LHR","time":"2021-07-26T17:38:19Z","message":"Connection dc4c2502-c0f5-4876-9f9e-6b3ea46c4607 registered"}
{"level":"debug","time":"2021-07-26T17:38:19Z","message":"rpcconnect: tx (finish = (questionId = 1, releaseResultCaps = false))"}

I have looked into your IP routes and you do not seem to have any route active at all.
I.e., if you run cloudflared tunnel route ip show, does it show anything? (it shouldn’t based on what I’m seeing).

If so, that’s the problem: check out https://developers.cloudflare.com/cloudflare-one/tutorials/warp-to-tunnel#create-a-tunnel ; that section has the steps that you are missing.

That’s weird, I followed those instructions before, but no routes were active. Must’ve deleted this while troubleshooting.

I re-enabled the route, but that hasn’t solved my problem.

$ cloudflared tunnel route ip add 192.168.2.0/24 9fa2b695-0306-4b27-abca-ff689ab5518d
$ cloudflared tunnel list

You can obtain more detailed information for each tunnel with `cloudflared tunnel info <name/uuid>`
ID                                   NAME                  CREATED              CONNECTIONS  
9fa2b695-0306-4b27-abca-ff689ab5518d tunnel-home-assistant 2021-06-23T09:15:27Z 2xAMS, 2xLHR
$ cloudflared tunnel info tunnel-home-assistant

NAME:     tunnel-home-assistant
ID:       9fa2b695-0306-4b27-abca-ff689ab5518d
CREATED:  2021-06-23 09:15:27.375814 +0000 UTC

CONNECTOR ID                         CREATED              ARCHITECTURE VERSION  ORIGIN IP      EDGE         
b17b5497-ea46-497c-b4d1-6db7788ce688 2021-07-26T17:38:17Z linux_amd64  2021.7.0 81.240.XXX.XXX 2xAMS, 2xLHR 
$ cloudflared tunnel route ip get 192.168.2.95

NETWORK        COMMENT TUNNEL ID                            TUNNEL NAME           CREATED              DELETED 
192.168.2.0/24         9fa2b695-0306-4b27-abca-ff689ab5518d tunnel-home-assistant 2021-07-27T15:36:14Z - 

In the meantime, I updated cloudflared to it’s latest version (2021.7.2).

None of my internal services are responding while connected through WARP. I don’t see any weird entries in the logs either. I do see good messages when I’m using my regular ingress (through subdomain on Cloudflare)

Edit: I do see some “errors” in the logs, but they don’t appear to be related to me connecting to WARP. In this case 192.168.2.187 is my cloudflared host and 192.168.2.95 is my Home Assistant instance. No errors reported about other services that I tried opening.

{"level":"debug","time":"2021-07-27T16:09:48Z","message":"CF-RAY: 6757225b6e61418a-AMS Request Content length unknown"}
{"level":"debug","time":"2021-07-27T16:09:52Z","message":"tunnel to origin copy: stream error: stream ID 365; CANCEL"}
{"level":"debug","time":"2021-07-27T16:09:52Z","message":"origin to tunnel copy: read tcp 192.168.2.187:51272->192.168.2.95:8123: use of closed network connection"}
{"level":"debug","time":"2021-07-27T16:09:52Z","message":"tunnel to origin copy: stream error: stream ID 359; CANCEL"}
{"level":"debug","time":"2021-07-27T16:09:52Z","message":"origin to tunnel copy: read tcp 192.168.2.187:51268->192.168.2.95:8123: use of closed network connection"}

A single tunnel can’t process both Warp Ingress rules and Network routing. When warp-routing: enabled: true is present settings such as no-tls-verify are ignored as cloudflared isn’t parsing the inbound requests just forwarding them to the origin.

I stand corrected… apparently it can. However no-tls-verify is an ingress only style rule.

Can you check out https://help.teams.cloudflare.com/ in the machine where you are running WARP (and trying to access your internal services)? What does it show?

1 Like

Everything seems to check out:


Can confirm, I was burnt a couple times before about this.

The rule needs to be added in the specific ingress rule:

tunnel: tunnel-home-assistant 
credentials-file: /root/.cloudflared/9fa2b695-0306-4b27-abca-ff689ab5518d.json

warp-routing:
  enabled: true

logfile: /var/log/cloudflared
loglevel: trace
transport-loglevel: trace

ingress:
  - hostname: xxxxxxxxx
    service: http://192.168.2.95:8123
    originRequest:
      no-tls-verify: true
      # OR
      noTLSVerify: true
  - service: http_status:404

https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/configuration/ingress#noTLSVerify


But, why are you fetching via http and adding the noTLSVerify option?

Hi Matteo!

I have two use cases for cloudflared:

  • To expose my Home Assistant instance to the web using a subdomain and regular HTTPS. That’s what the ingress rule is for. No need for no-tls-verify there.
  • To expose other services on my network (NAS, Router, Adguard, …) I want to use a split-tunnel with WARP. I though that I needed to enable no-tls-verify for the HTTPS endpoints that have self-signed certificates. But as you point out, that’s not true. Thanks!

I’m removing no-tls-verify from my config file. But the issue with WARP is still there.

Exactly. Once you route via IP Cloudflare doesn’t judge you (unless you add rules for it) for using HTTP. The other is HTTP directly.

You can add a valid certificate for it, there is an easy add-on that supports Cloudflare’s tokens for DNS challenges and allows internal HTTPS over a domain with full HTTPS end-to-end.

The WARP to Tunnel routing is purely TCP in cloudflared, so there’s no certificate verification issues there.

However, Gateway (that is in the middle) does TLS termination to apply L7 filtering rules. If the origin certificate cannot be validated, I suppose that can cause problems.

The current solution to this problem is to apply a “Do Not Inspect” rule to the private destination IPs that you know are going to apply such invalid certificates. See https://developers.cloudflare.com/cloudflare-one/policies/filtering/http-policies#do-not-inspect

Right, I totally forgot that.

@nuno.diegues Isn’t L7 too high up? I thought that traffic from a split-tunnel is not touched at all by Cloudflare, even if it’s HTTP-based.

Anyway, I finally solved my issue. I changed a few things but I have no idea what made it actually work.

  • I removed unused network interfaces from my Mac and restarted WARP afterwards just to be sure.
  • In Gateway > Network, I added a policy to block all traffic for my home network and only allow it for people with a certain email address. Found that advice here. But I highly doubt that this solved my problem.
  • I enabled the Proxy in Settings > Network > Proxy (Filter HTTP traffic at the Layer 7 firewall).
  • I deleted all IPv6 addresses from the split-tunnel configuration. IPv6 is not enabled within my home network, so this probably had no effect either.

I believe that my cloudflared instance was correctly configured and had a good connection to Cloudflare. The problem has to be somehow related to my Mac or Gateway.

Anyway, WARP split-tunnel is up and running. It’s an amazing service!

Thank you @nuno.diegues and @matteo for helping me debug this issue!

This was it.

It’s pointed out in this blue box: https://developers.cloudflare.com/cloudflare-one/tutorials/warp-to-tunnel#route-private-ip-ranges-through-warp
I didn’t remember to ask you to check that.

The split tunnel feature says which traffic goes to Cloudflare and which does not. For WARP traffic that goes to Cloudflare, and for users that are Teams’ enrolled, that traffic will go through Gateway (when you have Filtering enabled) and Gateway will perform L4 and L7 filtering.

Cloudflare Tunnel origins are only reachable via Gateway, so that’s why you had to enable Filtering and it started to work.

Glad to hear it is fine now!

2 Likes

I feel so silly for having glanced over that blue box… It’s embarrassing!

Thanks a ton!

Let’s not think about how many any of us has missed the obvious. It happens :slight_smile: