AAAA servers only accessible when Pseudo IPv4 enabled

I am only able to access my website using the Pseudo IPv4 option, as it causes NX_DOMAIN errors when disabled.

I noticed that when I disable the proxy setting, it does nothing to how the traffic is routed, as I can see that when I ping it goes to a CloudFlare endpoint. Maybe I am not waiting long enough for that to be corrected?

I am able to ping and view my website via IPv6 directly, so I am not sure what is preventing IPv6 traffic from hitting my servers from CloudFlare.

Thanks for any advice/input on this!

Pseudo IPv4 should not effect DNS at all. Could you share the domain? :slightly_smiling_face:

The domain is www.csacares.org

The domain resolves to both Cloudflare IPv4 and IPv6 addresses just fine on my end. Is your origin using IPv4 or IPv6?

It is dual stack, so it supports both. I should have also stated that this is in a split-horizon environment.

To add some more clarification:

  • I just noticed that I get 522 errors when trying to visit the website on the LAN side of the split.
  • I have set a CNAME for www to point to www.csacares.org.cdn.cloudflare.net on the LAN side of the split.

First, I’d like to say that I lack any knowledge about split-horizon setups - so please bear with me :slightly_smiling_face:

What are you trying to achieve by this? You should only CNAME to www.csacares.org.cdn.cloudflare.net if your Cloudflare zone is on a partial setup.

I was searching for this earlier and came across this:

That is my reason for using www.csacares.org.cdn.cloudflare.net

So I can confirm that CF is not sending any IPv6 traffic my way, and I cannot access my site from the CNAME as it give me 522 errors now.

A little bit more progress, but not out of the woods.

I have created a page rule that changes any URLs from *.csacares.org.cdn.cloudflare.net to *.csacares.org and that seems to force it over IPv4 a bit more reliably on the LAN, but I still see zero IPv6 traffic coming in, and will error out with a 522.

I can see that its failing over v6 because the IP address that is reported on the CF error page is always an v6 address. I never see a 522 error over ipv4 or connecting directly via ipv6.

There appear to be a large number of unrelated issues here. I may have missed a few.

With a dual stack Origin, Cloudflare still prefers IPv4 to the Origin, regardless of what the client is using. If you are having a multitude of issues, as you appear to be, it would probably be best to only use A records to the origin, and remove the AAAA records.

Where are you getting this error? Can you share a screenshot?

Is this from a client on the split brain DNS, or the public DNS? Is your internal DNS caching responses longer than the standard Cloudflare TTL? Do you have working IPv6 on your internal network (which is still pretty rare in corporate LANs)?

Do you mean entering www.csacares.org.cdn.cloudflare.net into the browser? That will not work, as there is no webserver configured with that name, it only has meaning in DNS.

Can you give a screenshot of this Page Rule?. No URL should ever actually reference that name, and unless you happen to control cloudflare.net (which you do not) I don’t see how you could create such a page rule.

Can you run the following command, and share the result? (Redact as needed to protect the Origin IP address)

curl https://[INSERT-ORIGIN-IPv6-HERE] -H "Host: www.csacares.org" -v -o /dev/null

This is just to set “fake” IPv6 addresses in the headers sent to your origin if your Origin webserver, logging system, etc. does not understand IPv6 addresses. Do you have HostnameLookups in your Origin config, or the equivalent for your webserver?

@Albert this is pretty normal in a split brain DNS setup. So long as the public Authoritative servers are set correctly as in a normal Full setup it works as expected.

1 Like

I did not know that CF prefers IPv4. However, I have removed the AAAA records and still get this issue. For what its worth, this only occurs in the region where my origin servers are located.

I was unable to recreate the NX_DOMAIN error, as I have been modifying my DNS configuration so much.

Sorry about the ambiguity, I am accessing the CNAME via the LAN side of the split brain via www.csacares.org as the DNS server resolves www.csacares.org to www.csacares.org.cdn.cloudflare.net.

LOL @ the page rule issue. I just realized that and removed that rule as looking back on it now does not make any sense.

It does not matter where I ping from, it always ends up at an IPv6 address (both www.csacares and cdn.cloudflare) when I have an A record setup. I have a functional IPv6 LAN as I am able to get to the website over IPv6 without using CF’s DNS.

Below is the output of the command you had me test:

curl https://\[IPv6_ADDRESS\] -H "Host: www.csacares.org" -v -o /dev/null
*   Trying IPv6_ADDRESS:443...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to IPv6_ADDRESS (IPv6_ADDRESS) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1080 bytes data]
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
} [2 bytes data]
* SSL certificate problem: self signed certificate
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I have had Pseudo IPv4 turned on and off and I get 522 errors either way. It seems that IPv6 is still how clients wish to connect, even though there is no longer a AAAA record.

That is normal. Cloudflare give both v4 and v6 proxy addresses by default. Most clients will use Happy Eyeballs, and will prefer functioning IPv6.

Do you use SSL Full or Flexible mode? If full, add -k to the command. If Flexible, change https to http. (And consider your life choices, and get a proper cert on your Origin and change to Full Strict!)

OK. If you are not relying on PseudoIPv4 leave it off as you don’t need it.

It looks like the only actual issue is the 522. (I get denied by your WAF rules, so cannot see this myself) Check out this Community Tip on how to troubleshoot 522 errors.

What country are you in? I can allow you in to test this out.

I use Flexible mode, but the certificate issue has never come up in the years I have been using CF. Using CF’s certificate service was the main reason why I moved in the first place.

EDIT: I have enabled Full, and still get the curl error. But I have copied the output of the second command, which looks a bit better.

 Connected to IPV6_ADDRESS (IPV6_ADDRESS) port 80 (#0)
> GET / HTTP/1.1
> Host: www.csacares.org
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Date: Wed, 01 Dec 2021 01:03:02 GMT
< Server: Apache/2.4.41 (Ubuntu)
< Strict-Transport-Security: max-age=31536000
< X-Frame-Options: SAMEORIGIN
< Set-Cookie: ppwp_wp_session=d346b70e6c7a7c3ebcfd8fc9e6b1d2c6%7C%7C1638322382%7C%7C1638322022; expires=Wed, 01-Dec-2021 01:33:02 GMT; Max-Age=1800; path=/
< X-Powered-By: W3 Total Cache/2.2.0
< Expires: Wed, 01 Dec 2021 02:03:02 GMT
< Cache-Control: max-age=3600
< X-Redirect-By: WordPress
< Location: https://www.csacares.org/
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy:
< Content-Length: 0
< Content-Type: text/html; charset=UTF-8
<
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host IPV6_ADDRESS left intact

On the LAN side of the split: I have reverted back to an IPv4 only configuration and I am still getting 522 errors. The website does not load on the first try, and I have to constantly reload the page to get it to work. I am not blocking port 80 or 443 access via .htaccess, iptables, or my firewall.

I am close to disabling the proxy to all my A records, as this issue is pointing to CF doing something weird with my web traffic.

EDIT 2: I have read the below guide, and it does not seem to cover my issue.

Here is the curl -k output with Full Encryption.

curl https://\[IPv6_ADDRESS\] -H "Host: www.csacares.org" -v -o /dev/null -k
*   Trying IPv6_ADDRESS:443...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to IPv6_ADDRESS (IPv6_ADDRESS) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [21 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1080 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=CA; L=Mountain View; O=Community Services Agency; OU=Tech; CN=www.csacares.org; [email protected]
*  start date: Apr 26 17:36:32 2018 GMT
*  expire date: Apr 23 17:36:32 2028 GMT
*  issuer: C=US; ST=CA; L=Mountain View; O=Community Services Agency; OU=Tech; CN=www.csacares.org; [email protected]
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
} [5 bytes data]
> GET / HTTP/1.1
> Host: www.csacares.org
> User-Agent: curl/7.68.0
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 01 Dec 2021 01:22:25 GMT
< Server: Apache/2.4.41 (Ubuntu)
< Strict-Transport-Security: max-age=31536000
< X-Frame-Options: SAMEORIGIN
< Link: <https://www.csacares.org/wp-json/>; rel="https://api.w.org/"
< Link: <https://www.csacares.org/wp-json/wp/v2/pages/11>; rel="alternate"; type="application/json"
< Link: <https://www.csacares.org/>; rel=shortlink
< Pragma: public
< Cache-Control: max-age=0, no-cache, s-maxage=10
< X-Powered-By: W3 Total Cache/2.2.0
< X-Mod-Pagespeed: 1.13.35.2-0
< Vary: Accept-Encoding
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< Referrer-Policy:
< Content-Length: 92839
< Content-Type: text/html; charset=UTF-8
<
{ [5 bytes data]
100 92839  100 92839    0     0  2590k      0 --:--:-- --:--:-- --:--:-- 2590k
* Connection #0 to host IPv6_ADDRESS left intact

Another update:

It looks like this also happens when I am trying to access any of my A records from California. I have made sure that I can directly access www by IPv4 address.

EDIT: I am not seeing any of these issues from Texas

It is looking like it was my IPv4 rules on my firewall. I thought I allowed only CF IP addresses, but obviously pulling the IP list from here is not propagating properly.

I will need to look into why my firewall rules are not working properly. I will keep this open for a while to make sure that is the case, but it looks like things are working a bit better after changing the rules.

Another update:

Still getting 522 errors on the LAN side of the split, but connections from California seem to be working now.

I have redirected my LAN segment to connect to the RFC1918 address of the web server. This was the only way I was able to get this to be reliable at all.

I have re-added the AAAA records for my domain, as I want to be able to have that ability to do so. Pseudo IPv4 is disabled, and even though I do not see any v6 traffic that is okay with me.

Bypassing the cache for my domain made things work a lot better. I also have setup Full Strict for SSL which seems to make a faster connection to the server.

At this point we are looking good. I will need to figure out if my firewall/NAT rules were causing any issues.

Thanks again to everyone who helped me out on this, as I obviously had put myself into a pretzel with my configuration.