Error 1001 for non-cloudflare domain pointing to a non-proxied load balancer

Performance
#1

Hello, I am working to configure an non-Cloudflare owned DNS record to point to a non-proxied Cloudflare loadbalancer.

I am attempting to configure dnstest.brinkman.mbb.sfu.ca to point at lb.sfu.brinkmanlab.ca via a CNAME record.
I have encountered an issue where it works for most of the public, except for some users that connect via a few large Canadian ISPs.
Those users encounter a Error 1001: DNS resolution error. I am aware of the “An external domain that is not on using Cloudflare has a CNAME record to a domain active on Cloudflare” policy, but I believe that is specific to proxied DNS records where a SSL certificate is provided by Cloudflare.

I attempted to configure “Custom Hostnames” in my brinkmanlab.ca zone, but encountered the issue that the CNAME record pointing at the loadbalancer domain conflicts with the required TXT record to validate the custom domain. A CNAME and TXT record can not exist for the same domain name. Regardless, I do not believe that configuring “Custom Hostnames” would resolve my issue as I am not routing via the Cloudflare proxy service.

An example of a user who encounters this error has an IP address of:

IPv6: 2001:569:7fce:f200:344a:9ecb:855c:dbfd
IPv4: 172.218.200.167

DNS: 75.153.171.67 75.153.171.116

I had another user provide a ray id: 6f7ce7d029e18450 2022-04-06 19:24:45 UTC

The error is clearly issued by Cloudflare, so it appears they are correctly resolving to the loadbalancer, it appears that the loadbalancer is not able to resolve the upstream destination. The only explanation I can think of is that these users are hitting a bad replica of the loadbalancer that is routed to via their ISP.

Can you please look into why the loadbalancer service is behaving this way?

Account id: 73335bb86fb1ec5a10267e19238d6407

Thanks

#2

https://support.cloudflare.com/hc/en-us/articles/360029779472-Troubleshooting-Cloudflare-1XXX-errors

#3

Thanks but I specifically addressed the content you linked to in my post.

#4

The community has no access to your account and cannot troubleshoot this win the way you expect. Have you already contacted support?

1 Like
#5

Support automatically closes the ticket insisting I post it here.

#6

Can you share any more diagnostic data here then, like traceroutes from the affected users?

Also please share your ticket number from support.

2 Likes
#7

Thanks @domjh for the help. I have been seriously scratching my head over this one.

After posting here I received a response to the closed ticket 2419435 from a rep that just repeated the bots response. I tried to reaffirm that something was wrong that is outside of my control, but have yet to get a response.

This morning I managed to get someone with access to a failing ISP, I had them run dig, traceroute and some curl tests:

traceroute -I dnstest.brinkman.mbb.sfu.ca
traceroute to lb.sfu.brinkmanlab.ca (206.12.7.220), 64 hops max, 72 byte packets
... internal network
4  64.114.6.147 (64.114.6.147)  7.111 ms  5.073 ms  5.027 ms
5  606-tx-sfu1-vncv1-cr2.vncv1.bc.net (134.87.30.241)  5.246 ms  5.062 ms  5.163 ms
6  van-hcc1360-f3300e-1-van-hcc1360-border-1.net.sfu.ca (142.58.254.168)  5.271 ms  4.841 ms  5.036 ms
7  van-int-sfu-ext-van-bcnet-int-sfu.net.sfu.ca (142.58.254.143)  5.007 ms  5.400 ms  5.523 ms
8  van-hcc1360-core-1-van-hcc1360-f3300e-1.net.sfu.ca (142.58.254.171)  5.988 ms  5.538 ms  6.253 ms
9  bby-wtb115-core-1-van-hcc1360-core-1.net.sfu.ca (142.58.29.9)  8.648 ms  6.722 ms  5.782 ms

curl -ivv http://dnstest.brinkman.mbb.sfu.ca --output /dev/null
* TCP_NODELAY set
* Connected to dnstest.brinkman.mbb.sfu.ca (2606:4700:3033::6815:1339) port 80 (#0)
> GET / HTTP/1.1
> Host: dnstest.brinkman.mbb.sfu.ca
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 409 Conflict
< Date: Fri, 08 Apr 2022 17:20:59 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 16
< Connection: close
< X-Frame-Options: SAMEORIGIN
< Referrer-Policy: same-origin
< Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Expires: Thu, 01 Jan 1970 00:00:01 GMT
< Server: cloudflare
< CF-RAY: 6f8cad410e918419-YVR
<
{ [16 bytes data]

curl -ivv -H 'Host: dnstest.brinkman.mbb.sfu.ca' http://206.12.7.220 --output /dev/null

* TCP_NODELAY set
* Connected to 206.12.7.220 (206.12.7.220) port 80 (#0)
> GET / HTTP/1.1
> Host: dnstest.brinkman.mbb.sfu.ca
> User-Agent: curl/7.64.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< date: Fri, 08 Apr 2022 19:18:37 GMT
< server: Apache/2.2.3 (Linux/SUSE)
< x-powered-by: PHP/5.2.0
< x-pingback: https://www.brinkman.mbb.sfu.ca/wordpress/xmlrpc.php
< transfer-encoding: chunked
< content-type: text/html; charset=UTF-8
<
{ [2590 bytes data]
100 11620    0 11620    0     0  46854      0 --:--:-- --:--:-- --:--:-- 46854
* Connection #0 to host 206.12.7.220 left intact
* Closing connection 0

dig dnstest.brinkman.mbb.sfu.ca +showsearch

; <<>> DiG 9.10.6 <<>> dnstest.brinkman.mbb.sfu.ca +showsearch
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23326
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;dnstest.brinkman.mbb.sfu.ca.    IN    A

;; ANSWER SECTION:
dnstest.brinkman.mbb.sfu.ca. 3600 IN    CNAME    lb.sfu.brinkmanlab.ca.
lb.sfu.brinkmanlab.ca.    10    IN    A    206.12.7.220

;; Query time: 15 msec
;; SERVER: 2001:568:ff09:10c::68#53(2001:568:ff09:10c::68)
;; WHEN: Fri Apr 08 10:22:42 PDT 2022
;; MSG SIZE  rcvd: 105

206.12.7.220 is the correct origin IP, so it seems that every diagnostic is succeeding except for a HTTP request. I almost wonder if they are hitting a loadbalancer replica that errantly is trying to pass the request through the CF proxy. I have triple checked that no proxy is enabled.

They’re using OSX with a gimpy traceroute that doesn’t seem to support TCP SYN packets. Here is my traceroute on a ISP that does not have this issue:

traceroute -T dnstest.brinkman.mbb.sfu.ca

traceroute to dnstest.brinkman.mbb.sfu.ca (206.12.7.220), 30 hops max, 60 byte packets
...internal network
 9  64.251.87.210 (64.251.87.210)  24.421 ms  23.128 ms  23.143 ms
10  606-tx-sfu1-vncv1-cr2.vncv1.bc.net (134.87.30.241)  25.186 ms  23.635 ms  25.124 ms
11  van-hcc1360-f3300e-1-van-hcc1360-border-1.net.sfu.ca (142.58.254.168)  60.247 ms  60.161 ms  22.055 ms
12  van-int-sfu-ext-van-bcnet-int-sfu.net.sfu.ca (142.58.254.143)  22.009 ms  21.818 ms  25.099 ms
13  van-hcc1360-core-1-van-hcc1360-f3300e-1.net.sfu.ca (142.58.254.171)  24.264 ms  25.962 ms  25.648 ms
14  bby-wtb115-core-1-van-hcc1360-core-1.net.sfu.ca (142.58.29.9)  27.596 ms  25.866 ms  26.802 ms
15  res-proxy.its.sfu.ca (142.58.29.73)  26.164 ms  22.831 ms  21.195 ms
16  bby-wtb115-a7050-l1-2-bby-rsch-vdom.net.sfu.ca (142.58.29.51)  26.627 ms  26.202 ms  26.284 ms
17  * * *
18  * * *
19  rcg-lb-2.dcr.sfu.ca (206.12.7.220)  46.646 ms  45.343 ms  45.918 ms
20  rcg-lb-2.dcr.sfu.ca (206.12.7.220)  45.490 ms  38.590 ms  48.457 ms
#8

Thanks for all the info, that is indeed odd. I’ve added this to the community escalations but it might now be Monday before those are looked at - hopefully you get a response on the ticket before then.

1 Like
#9

Thanks again for the help, CF support responded.

It looks like the issue is because I have the loadbalancer pointed to an origin without an AAAA record. The failing ISPs are issuing their clients IP6 only. When the loadbalancer cant resolve an AAAA record during the request it emits Error 1001 rather than failing over to IP4.

The engineers are going to look into this.

image

1 Like
#10

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.