Desktop traffic sometimes passing through Warp when using 1.1.1.1 for DNS?

Recently, we were looking through our access logs to troubleshoot a bug and noticed a number of anomalous requests from IP addresses operated by Cloudflare.

We use realip with Nginx, so we typically expect to pull the real IP address of the end user from the last trusted element of the X-Forwarded-For header. We also log both X-Forwarded-For and CF-Connecting-IP separate from the IP address that realip chooses.

Because of this configuration, we typically only expect to see Cloudflare IP addresses when our site is visited by the AlwaysOnline bot or Cloudflare staff members. However, our logs showed requests that didn’t fit into either of those categories starting in September.

When we took a closer look at the requests, we noticed that:

  • They all originated from desktops, not mobile devices, indicating that Warp wasn’t in use. Exclusively desktop–zero mobile requests.
  • The user agents often indicated that a user was likely technically inclined (e.g., Linux).
  • When the IP addresses were IPv6, they were often in 2a09:bac0::/32. Despite being announced by AS13335 (Cloudflare), that range isn’t in Cloudflare’s list of IP addresses that they operate, which makes sense for IP addresses used by Warp.
  • A WHOIS query on the allocation (RIPE) shows CLOUDFLAREWARP in the netname field.
  • Often requests would switch between residential IP addresses and Cloudflare IP addresses with zero delay. A series of requests would come in from a session on a residential IP address, and within milliseconds, more requests would come in from the same session, but on a Cloudflare IP address. It would later switch back, again with zero delay.
  • X-Forwarded-For and CF-Connecting-IP both showed the anomalous Cloudflare IP addresses. There was never an additional residential IP address in X-Forwarded-For.

We contacted one of our users whose requests were seen coming from Cloudflare IP addresses. They informed us that they have 1.1.1.1 configured as the DNS server in their router. Based on further investigation and analysis, we believe most or all of the anomalous requests originated from people who were using 1.1.1.1 as a DNS server but were not using Warp.

When people use Warp from a mobile client, we see their real IP address without issue. We haven’t found any cases in which we saw a Cloudflare IP address in place of a real IP address for a Warp user. This only seems to affect people using 1.1.1.1 for DNS, but not as a VPN.

This leaves us with several questions:

  1. Why are Warp IP addresses sometimes being reported in place of real IP addresses when people aren’t actually using Warp?
  2. Why does this seem to affect people who are using 1.1.1.1 as a nameserver, but not as a VPN?
  3. How is this even possible? It doesn’t make sense that Cloudflare’s HTTP servers would be aware of whether someone is using 1.1.1.1 as a DNS server. Is this a bug that can affect any request, and the correlation with 1.1.1.1 just a crazy coincidence, or is 1.1.1.1 sometimes serving different A/AAAA records than would otherwise be expected?

Here’s an example request from two days ago:

  • CF-Ray: 545d10ae8dcce81d-LAX
  • User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0
  • X-Forwarded-For: 2a09:bac0:12::82f:4597
  • CF-Connecting-IP: 2a09:bac0:12::82f:4597
  • IP address that actually made the request (typically the Cloudflare PoP): 172.68.46.239

So you can absolutely confirm these requests were sent from a desktop system (not e.g. a mobile system posing as desktop browser) and - apart from your site being on Cloudflare - the only Cloudflare related thing there is (on the client side), is the common configuration of using Cloudflare’s public resolver.

Would that be an accurate summary?

That would mean Cloudflare would supposedly route some(?) requests not (just) via the proxies but also via the VPN system. Somehow that sounds improbable, however I would say that would be a case best for a support ticket. Support should hopefully be able to shed some light here. That would not be a setup I’d expect in any way.

Could it be possible that CF is doing some A/B testing if you are using 1.1.1.1 and if your request falls in that % who get the experimental treatment, they get sent to a different CF ingress IP which randomly makes them pass through the Warp network?

Yes, that appears to be the case.

Very improbable. I don’t understand how it could happen from a technical perspective–maybe the correlation is just a coincidence. I thought maybe 1.1.1.1 was occasionally returning A/AAAA records for Warp endpoints, but I tried manually making requests through such endpoints (as if I had been served such A/AAAA records), and was unable to get through on port 443. (This was all HTTPS traffic.)

I’m hesitant to open another support ticket. We discovered this while assessing the impact of what we believe to be a very serious Cloudflare vulnerability for which we already have an open ticket (1800313). Unfortunately, we’ve received no communication on that ticket even though it’s been open for nearly two weeks–we just keep getting, “our engineering team will get back to you”, “it’s been prioritized”, etc. This is the first time we’ve ever had trouble getting a support ticket resolved, and I’m concerned that opening another related ticket (or even replying with this info in the existing ticket) will delay the resolution of the far more serious issue.

I don’t think so; the percentage of requests that exhibit this anomaly is far, far too small. Some quick napkin math says it affects very roughly somewhere between 1 in 1M to 1 in 100M requests. We have staff who use 1.1.1.1 and/or Warp; despite continual requests to our site every day, it has never affected any of our staff.

Most of the affected requests appear to originate from PDX and LAX, so it’s possible it’s a bug that only affects those datacenters.

That would actually be plausible, considering the scale of CF, it’s entirely possible that they are testing some new feature/product/stack on an extremely small sample set so they wouldn’t get drowned in requests to a system that may or may not work.

I believe they have canary datacenters that they use for testing. If what you’re suggesting is true, then the canary datacenters are PDX and LAX, at least for this. That’d mean it’d have to affect some percentage of traffic going through those datacenters. However, we have a lot of requests coming from LAX, so it’s still an oddly small number. In the past, when we’ve noticed issues that affect numbers like these, it’s been the result of a bug.

Even if that were the case, Warp normally passes the real IP address in CF-Connecting-IP and X-Forwarded-For, so it’d still be a bug.

Another problem with that theory: why would it affect one user for X amount of time, leaving them unaffected immediately before and immediately after? Why does X vary so widely? (It might be a minute, it might be an hour.)

Well, if the observed behaviour is accurate, something like that would need to be the case. Either they return a non-proxy address on a DNS level or the proxies do not proxy to the origin but to the VPN platform instead.

@cloonan

You mean because you have two tickets in parallel? Shouldnt be the case, but of course you never know, though the first ticket doesnt seem to be in a hurry anyway :wink:

@cloonan might have best a look at the security related ticket, but I am afraid for the issue at hand here support is most likely also the best contact, as the described scenario would be very unusual and only Cloudflare could tell what might be going on here. Even if it is @Vilsol’s theory of some test, that would be only something Cloudflare could confirm or deny. The community most likely wont be able to help here much. You could only mass tag Cloudflare staff :slight_smile:

1 Like

It might be that just being pointed to a canary IP by 1.1.1.1 is not enough and the node itself splits the incoming traffic. That would also explain why making manual requests to the NS didn’t produce different results, as it always returns the canary IP range?

It still doesn’t explain why the headers contain the CF IP and not the forwarder-for one.

More because they’re related. We did have an open ticket in parallel, but it was resolved quickly–as was always the case prior to this serious ticket. Cloudflare has excellent support, even for small customers–and we really do appreciate it. I’m always afraid to open unnecessary tickets because I don’t want to add too much of a burden. This really wouldn’t be much of a concern if it weren’t making it difficult for us to assess the impact of the security issue; we’d normally just tag the anomalies in our logs and shrug it off.

Darn; I was hoping maybe somebody had noticed something similar and had already found an explanation. Support ticket it is!

I have Warp on my phone, so I grabbed some of the random endpoints in the logs and tried to make requests through them via cURL. However, I made the mistake of doing a preliminary test: I ran curl https://162.159.192.3/, saw an SSL alert (sslv3 alert handshake failure), and gave up. However, it just occurred to me that it might’ve been mad because I wasn’t passing anything via SNI, so I tried again with curl --resolve cloudflare.com:443:162.159.192.3 https://cloudflare.com/cdn-cgi/trace -v–which works but doesn’t show Warp as being in use. Which is when I realized that 162.159.192.3 is just a standard Cloudflare PoP IP address and appears in their IP list–it’s not a Warp IP address. Whoops.

Thank you, all. I see the ticket and related issues. The Support engineer is pushing this with the team and I’ll add myself to the ticket.

Edit - I talked with the Engineer on the ticket and they’ve escalated, will continue to monitor.

2 Likes

Of course I cant speak for the entire community, maybe there will be eventually someone who has observed the same behaviour. So far the issue hasnt been mentioned by anyone and thats why I believe a support ticket will get you better results. I might be wrong :smile:

2 Likes