Cloudflare and Nginx Connection Error 522

Website, Application, Performance
1

I will monitor this thread to the best of my ability and any guidance or help is greatly appreciated.

So after a period, I am reaching out to the community. We had severe weather a few weeks ago and I shut my server down as usual but when I brought it back up all of my websites were down. I create A records in Cloudflare and Nginx for all my applications I access on other devices like Overseer, Portainer, and Nginx. These domains all worked with no issues prior to the shutdown and when bringing them back up they stopped working. I checked the server and nothing appeared to change when the system came back. Since coming back I have only gotten Cloudflare Error 522. I am having difficulty digesting and determining the issue and route cause. Initially, I thought it was an SSL authentication issue between Cloudflare and Nginx but again nothing there changed. Just a quick disclaimer, none of these sites are working. I had a total of 6 of them so I don’t believe it is as simple as a specific setting for a docker container.

Cloudflare:

  • A record for Nginx.example.com with my local IP, this record is proxied.
  • Force HTTPS enabled
  • SSL/TLS encryption mode is set to Full (tried Full (strict) and it didn’t work either with any testing)
  • API token for all zones created for Nginx SSL Cert.
  • I do not have the pro-plan so I cannot simply input a ticket for help from Cloudflare.

Nginx:

  • Source: domain created in Cloudflare.
  • Destination: At this moment it is set to //127.0.0.1:81 or //localhost:81
    • Side note, This is something I am on, I have watched many videos and am unable to determine which IP is used, I have watched some use their LAN or the systems’ IP, some use the IP used to connect locally, and a few other variations. This is the IP Nginx uses to connect to
  • SSL Cert:
    • I used a wildcard cert (*.example.com) pointed this at Cloudflare and provided the API token from Cloudflare. I used the curl command to confirm this cert is valid and working.

Portainer: version 2.18.2

  • At this moment using nginx image: jc21/nginx-proxy-manager:latest.
  • Published ports 80:80, 81:81, 443:443 in a container and forwarded on ISP Router.
  • I have tried numerous things, I started from scratch on the image, I tried defining my domain under the network section, and published ports, We have been trying the same with Overseer to try and get anything to work and even disabling all SSL to the best of our ability and trying to make an unsecured connection to the webpage has not worked, continues to give the 522 error.

Questionable items:

  • When researching and gathering information I noticed services like Overseer running on IP 10.0.0.208:Portnumber whereas Nginx is running on 127.0.0.1, I have not dictated the IPs for these containers, and in Portainer I do not have their part of a specified network right now.
  • When going through the port forwarding setup on my ISP router I found that it reserved an IP and in my app, it shows my server IP as 10.0.0.208 but nowhere else does this show or reflect and the connected gig ethernet NIC has the IP of 192.168.x.x

System:

  • Ubuntu 22.04.2 LTS
  • i3-9100F, AMD Radeon RX 560, 16GB RAM, RealTek Gig/Ethernet NIC, 40TB.
4

Has your public IP been rotated by the ISP? You mentioned you had severe weather and you shut down your server. I would imagine your modem/gateway may have been shut off as well. ISPs don’t keep IP addresses reserved for you. If your modem is offline for more than about an hour or two, they’ll usually release it and give you a new one.

5

That was the first thing I checked, I have a reserved IP so it stays the same.

6

OK. I just wanted to cover all the bases.

When I asked that question, I could not ping the IP address in your A records.

As of this comment, now:
I can ping you from my laptop, on and off my Cloudflare VPN.
Cloudflare can ping you from the edge servers.

I can curl your IP address through my Cloudflare VPN and see the connection open up but nothing is sent back before I get a 504:

(IP obviously replaced for privacy)

$ curl -svo /dev/null W.X.Y.Z
*   Trying W.X.Y.Z:80...
* Connected to W.X.Y.Z (W.X.Y.Z) port 80 (#0)
> GET / HTTP/1.1
> Host: W.X.Y.Z
> User-Agent: curl/8.1.2
> Accept: */*
>
< HTTP/1.1 504 Gateway Timeout
< Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Referrer-Policy: same-origin
< Expires: Thu, 01 Jan 1970 00:00:01 GMT
< Proxy-Status: Cloudflare-Proxy;error=connection_timeout
< Content-Length: 10164
< Date: Mon, 28 Aug 2023 20:07:07 GMT
<
{ [786 bytes data]
* Connection #0 to host W.X.Y.Z left intact

Cloudflare’s edge servers cannot curl your origin at all:

*   Trying W.X.Y.Z:443...
* connect to W.X.Y.Z port 443 failed: Connection timed out
* Failed to connect to W.X.Y.Z port 443 after 15256 ms: Couldn't connect to server

If I try to directly curl your origin without my Cloudflare VPN enabled, it times out completely:

$ curl -svo /dev/null W.X.Y.Z
*   Trying W.X.Y.Z:80...
* connect to W.X.Y.Z port 80 failed: Operation timed out
* Failed to connect to W.X.Y.Z port 80 after 75006 ms: Couldn't connect to server

$ curl -svo /dev/null W.X.Y.Z:443
*   Trying W.X.Y.Z:443...
* connect to W.X.Y.Z port 443 failed: Operation timed out
* Failed to connect to W.X.Y.Z port 443 after 75005 ms: Couldn't connect to server

This looks like you may have configured things to only allow Cloudflare IPs to your webserver (which is fantastic), but we are trying to make the connection and nothing is being served back.

Possibilities:

  • Server DHCP IP changed, invalidating any port forward.
  • NGINX didn’t come back up properly.
  • Invalid NGINX Configuration file.
  • Hardware failure (LAN cable from router/switch to server, etc.)
  • Server firewall settings running in memory but not saved to re-implement at boot.
1 Like
7

I am an entry level networking student, and I am trying to narrow this down. On my end when I do a curl, it gives a 301 Moved permanently. The port forwarding done on my ISP router is only port based not IP based, so unlike pfsense where you can specify any HTTP traffic coming into port 80 forward to w.x.y.z it only asks what device and the port. I have checked the firewall and like I said it wasn’t a config file issue because none of the websites worked not just nginx.

I feel like it could be some sort of configuration, but I am not sure how to diagnose this further. Any pointers or tips you might be able to provide? For instance, my first thought was possibly the incoming requests and outgoing requests to/from Cloudflare to/from Nginx are not able to make the connection. This was why I checked the IP first but like I said in my post it seems some of the passwords when doing ifconfig don’t match what I expected but the local hosts addresses running off these numbers work and nothing seems to have changed. But we have tried using the other IPv4 addresses, and they don’t work either.

The strangest part is yes, we restarted the system but I have done this prior and no issues, nothing changed or updated that I can find, my plex still works and all is well there, but we even started nginx fresh my removing the old containers and images and following a guide that we have used before but not the same image, to create the container and it got the same error.