Cannot access Portainer instance via Cloudflare Tunnel getting 502 error

Am new to Cloudflare and started exposing my services to internet via Cloudflare tunnels. Am setting all this with Raspberry Pi running Docker and maintaining all container with application Portainer. What am trying to achieve is via Cloudflare tunnel securely can access Portainer GUI so can maintain my containers remotely. Did successfully had application “Nextcloud” via Cloudflare tunnel and am able to access it remotely using hostname set up during tunnel configuration of Nextcloud Instance but same is not working for Portainer. Always getting stuck with Error 502 Bad Gateway. Did someone face any issue regarding the same please help.

Hi,

Thank you for asking.

May I ask if you’re trying to achieve to have it like portainer.example.com to access Portainer GUI, which is a proxied :orange: hostname configured as a CNAME for the cloudflared tunnel?

May I ask if you’re using some reverse proxy like nginx proxy manager or traefik, or some other, to run Portainer?

Is Portainer GUI running on a local port 9000 as default or some different?

Despite the usage of the cloudflared tunnel, can you successfully access the Portainer GUI without it?

Is that hostname covered with an SSL or not and is it working over HTTP or HTTPS?

How does your cloudflared configuration file look like?

Just in case, despite it’s 502, regarding the Docker, may I ask if Cloudflare IPs are allowed?

Hi Fritex,

First of all thank you for your help. Really appreciate that. Below are my answers to your questions.

Have purchased my domain name from Cloudflare so Yes, am using for example 'xxxxxx.mydomain.com' to access Portainer GUI. Yes, my DNS setting in cloudflare webpage is displayed as xxxxxx as CNAME for the cloudflare tunnel and am suing default setting so Yes, it is proxied with orange cloud icon of cloudflare. 

No, am not using any reverser proxy like nginx proxy manager. Was actually using it but have deleted all the proxies and it is empty now. But Yes, do have Nginx Proxy Manager Container running in my Portainer Instance. Did try with disabling Nginx proxy manager as well but that didn't help.

No, Portainer GUI after recent upgrade have moved to port 9443 https by default. So am using my raspberrypiIP:9443 to access Portainer GUI locally.

Yes, without cloudflare tunnel in my local home network as said am accessing portainer GUI as raspberrypiIP:9443 and it works perfectly.

As said above that as now the Portainer instance by default has moved to port 9443 so it does runs with https but there is no SSL so it gives me that warning "not secure" while browsing to that raspberrypiIP:9443

For cloudflared configuration: Basically started off with creating tunnel with basic settings following step by step on screen instructions by cloudflared. When it asked for Public Hostname gave 'xxxxxx.mydomain.com'. So here basically as you know xxxxxx this is one we select of our own and by default it takes mydomain.com as xxxxxx will be CNAME record for my main domain mydomain.com

Am not sure about your last question but yes my tunnel is up and running and while in raspberrypi terminal when am pinging 'xxxxxx.mydomain.com' it pings successfully. 

As part of troubleshooting am thinking if there is any DNS record keeping file for Portainer Instance such where we have to update that we are going to access Portainer using some domain name such as the case with Nextcloud. Because am remembering had to update the config.php file of Nextcloud and add that domain name for nextcloud that will be accessing through that domain name. So not sure if there is similar thing for Portainer. 

Let me know if there are any more questions and we can soon find the root cause for this. 

Thanks again for your help. Looking forward for your reply.

Hey @uttarms, I am sorry to write back a bit late, however that’s weird to me :thinking:

Can we reproduce the steps and check the config.yml for cloudflared?

What I’ve done in the meantime and it’s still working via cloudflared tunnel over Docker and accessible to public via portainer.example.dev.

  • I installed Docker on my Raspberry Pi 4 which is connected to my local home network.
  • Added my user to docker group
  • Just in case, I’ve configured my custom RPi hostname and added my ISP IP address as rpi.example.com into the hosts file
  • Created Portainer volume docker volume create portainer_data
  • Run the docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
  • Logged-in and created admin user via RPi Web browser via https://localhost:9443 (HTTP only working)
  • Downloaded and installed armhf.debRelease 2022.10.1 · cloudflare/cloudflared · GitHub
  • Then authenticated as cloudflared tunnel login in my Web browser for my domain
  • Created cloudflared tunnel as cloudflared tunnel create portainer
  • Inside /root/.cloudflared/config.yml, created and saved config.yml which contains basic:
tunnel: CLOUDFLARED-TUNNEL-ID
credentials-file: /root/.cloudflared/COUDFLARED-TUNNEL-ID.json

ingress:
 - hostname: portainer.example.dev
   service: https://localhost:9443
   originRequest:
     connectTimeout: 30s
     noTLSVerify: true
 - service: http_status:404
  • Started traffic routing as cloudflared tunnel route dns portainer portainer.example.com
  • Run the tunnel as cloudflared tunnel run portainer

NOTE: For example.dev and portainer.example.dev in my case, I use Full (Strict) SSL at SSL/TLS tab.
Despite the portainer.example.dev, for the example.dev (naked domain) I am using “Cloudflare Origin CA Certificate” and “Authenticated Origin Pulls” → doesn’t matter, but just to mention.
Local RPi doesn’t have SSL certificate, running only on HTTP.
However, in my config file it has https://localhost:9443.
Therefore, the noTLSVerify parameter set to true.
And default service http_status:404 at the end.
CNAME tunnel hostname is proxied :orange:.

Working for me, CNAME was created via cli:

On my local desktop PC via local ISP:

Portainer interface visible and running on the localhost on my RPi4 via RDP on my Windows OS and also in my Web browser via portainer.example.dev. In the background can be seen the Terminal running on my RPi4 too.

On my mobile phone via mobile data (4G LTE network):

Questions:
I wonder if we’ve missed some step in your case :thinking:
Otherwise, maybe the ISP is blocking something, like port on the device (router), etc.?

Helpful resources which I’ve checked and followed and achieved the same:

1 Like

Hi Fritex,

Once again thank you very much for responding back. Appreciate man! Looked at all your step. But here is what fixed the issue but still wanted to discuss more and see that if it is the right/secure way we should access portainer over Cloudflare tunnels. Am only answering below which is slightly different in my case.

Answer to point 3 : Did not happen to add this IP : Hostname in my hosts file in raspberrypi. Maybe my A Record on cloudlflare pointing my domain to Server IP have taken care of for this step.
Answer to point 6: In my case used server IP where portainer is running i.e. serverip:9443

Now here is how it started working and as we guessed it has to do with SSL/TLS setting. While configuring tunnel there is a tab for “Additional Application Settings” in which there is “TLS” setting

Hope you can see the image above. This is the place which created the issue. When this toggled to ON meaning Enabled , was able to access portainer via my domain portainerdotmydomaindotcom.

Reading on the docker portainer installation it says they have now forced portainer to work on port 9443 with its own self signed certificate (SSL). With this no TLS verify: Disables TLS verification of the certificate presented by your origin. Will allow any certificate from the origin to be accepted. My understanding is when we are going through tunnels Cloudflare is generating its own SSL certificate for us so enabling this option in specific to portainer is ok. Please let me know what are your thoughts on this.

So are you also accessing the portainer via same TLS setting?

Also my approach is to create tunnels through docker command so it creates a separate container which am maintaining it through portainer. So basically when certain application is not needed to be available online am just bringing down the tunnel connector container and then the application is not reachable online. Meaning, use application whenever needed with bringing tunnel up through bringing up the container. Was expecting there was a way to shut down or bring up tunnel on Cloudflare itself.

Thanks once again, looking forward for your reply!

1 Like

Hi @fritex ,
I’ve the same issue but it’s weird. Some tunnels with subdomain does working well, but the root domain (which is my wordpress blog) is not. I’ve tried to use the same running tunnel but no hope with 502 error. My logs:

2022-11-13T15:27:42Z INF Starting tunnel tunnelID=3336afb3-deea-42d9-ae29-cd8407a62d8b
2022-11-13T15:27:42Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
2022-11-13T15:27:42Z INF Version 2022.10.3
2022-11-13T15:27:42Z INF GOOS: linux, GOVersion: go1.19.2, GoArch: amd64
2022-11-13T15:27:42Z INF Settings: map[no-autoupdate:true token:*****]
2022-11-13T15:27:42Z INF Generated Connector ID: ea38d5ba-c72f-4252-8b95-16ac5cca59a2
2022-11-13T15:27:42Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
2022-11-13T15:27:42Z INF Initial protocol quic
2022-11-13T15:27:42Z INF ICMP proxy will use 172.17.0.3 as source for IPv4
2022-11-13T15:27:42Z INF ICMP proxy will use :: as source for IPv6
2022-11-13T15:27:42Z INF Starting metrics server on 127.0.0.1:42267/metrics
2022/11/13 15:27:42 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022-11-13T15:27:43Z INF Connection 642d03cd-9d02-4659-b842-215f85a9017e registered connIndex=0 ip=198.41.200.193 location=SIN
2022-11-13T15:27:44Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"example.com\",\"originRequest\":{\"noTLSVerify\":true},\"service\":\"https://192.168.0.127:8081\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=3
2022-11-13T15:27:44Z INF Connection bee32659-6187-4e4f-abe4-2e9903757711 registered connIndex=1 ip=198.41.192.57 location=HKG
2022-11-13T15:27:45Z INF Connection b3ca2718-4169-4ef8-89d7-61a743f64ce8 registered connIndex=2 ip=198.41.200.43 location=SIN
2022-11-13T15:27:46Z INF Connection 84b89c8c-71f3-465c-b362-46fb56c8baf2 registered connIndex=3 ip=198.41.192.7 location=HKG
2022-11-13T15:28:35Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: first record does not look like a TLS handshake" cfRay=769889bb8bf7044b-HKG ingressRule=0 originService=https://192.168.0.127:8081
2022-11-13T15:28:35Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: first record does not look like a TLS handshake" connIndex=1 dest=https://example.com/ ip=198.41.192.57 type=http
2022-11-13T15:28:35Z ERR  error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: first record does not look like a TLS handshake" cfRay=769889bd8e9e044b-HKG ingressRule=0 originService=https://192.168.0.127:8081
2022-11-13T15:28:35Z ERR Request failed error="Unable to reach the origin service. The service may be down or it may not be responding to traffic from cloudflared: tls: first record does not look like a TLS handshake" connIndex=1 dest=https://example.com/favicon.ico ip=198.41.192.57 type=http
1 Like

@long.hd, could below article maybe help in your case? :thinking:

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.