One Tunnel for Multiple Servers with Different Services

[waleed-alharthi]

I’ve registered and configured my domain’s name servers to use Cloudflare.

I’ve configured a bunch of web apps on “Server A” to run through a Cloudflare tunnel. Everything works beautifully, and I can access my apps on subdomain.domain.whatever without problems.

I’m using docker compose to run the tunnel service:

  tunnel:
    image: cloudflare/cloudflared
    restart: always
    command: tunnel run
    environment:
      - TUNNEL_TOKEN=my_token_here

Then exposing my http services to the public hostname using the online dashboard.

Now, my challenge is when trying to run a copy of “my-service”, or any service (not necessary replicas) on “Server B”, this server can be local or remote.

I’m using the exact same tunnel configuration locally.

The tunnel service is running on the same docker network as the services I’m trying to expose.

The services themselves are healthy and accessible on localhost or through port forwarding.

On Cloudflare, both servers “Connectors” appear under the same tunnel

Whenever I try to access anything from “Server B”, I run into the infamous error message “Host Error”

Update 1: Interesting findings today, either everything on “Server A” works, or everything on “Server B”, but never both. Like a failover for the whole server, but not for services within the server.

In the example below I can access either:

Services A, B, and C, or

Services C, D and E.

but never A, B, C, D, and E.

[colinbarr]

As you’ve seen, it’s possible to deploy multiple instances (connectors) attached to the same tunnel by running cloudflared more than once with the same token. This mode is intended to provide high availability/failover, where each connector has equivalent network connectivity - which doesn’t look to be the case here. There’s no guarantee, however, which of the connectors is used at any given time, which would explain the flapping behaviour you’ve seen. That also means there’s no automatic affinity to a particular connector, just because cloudflared happens to be running adjacent to the service that’s being accessed.

There’s two different ways you could solve this:

1. Create and use separate tunnel for the cloudflared instances on server B. This is likely the most straightforward approach with the least changes.
2. Use hostnames and/or port mappings for your services that would be be accessible from either server A, or server B (so a request may tunnel through cloudflared on server B, but it’s actually for server A). In this case, you would have something like http://server-a.example.com:1234 for service A, http://server-a.example.com:1235 for service B etc., and as long as that was accessible from either of your servers it would work. This option is based on an assumption that you’ve currently got your tunnel config set with localhost and the exposed port, or the internal Docker network hostname, which is why A<>B isn’t working.

[waleed-alharthi]

Many thanks @colinbarr for the quick turnaround.

In terms of checking for a healthy server, the tunnels work fine; if the whole server is down, uses the other one.

I was hoping for an approach where i can use the same url/port and get directed to the healthy service.

If I use a separate tunnel for “Server B”, the immediate issue will be fixed but I’ll lose high availability for replicated services (unless I give them different URLs, which won’t keep my end users happy).

The second approach is more like what I’m after (so long as I don’t need to change the URL at the client side). I am using the same name for the docker network name on both machines but maybe they’re treated as 2 separate networks?

I use the exact same network name, service name and port on both servers hoping to map to a single URL. How’d I configure this page if the services had different ports? Or is there no escape from having different URLs or ports?

Another approach I thought about is running a tunnel for each stack (services that run together), but I’d rather keep things simple.

[colinbarr]

@waleed-alharthi

Many thanks @colinbarr for the quick turnaround.

No problem, happy to help.

@waleed-alharthi

In terms of checking for a healthy server, the tunnels work fine; if the whole server is down, uses the other one.

I was hoping for an approach where i can use the same url/port and get directed to the healthy service.

If I use a separate tunnel for “Server B”, the immediate issue will be fixed but I’ll lose high availability for replicated services (unless I give them different URLs, which won’t keep my end users happy).

This is the intended behaviour of running multiple tunnel connectors (HA for the connectors), but without using a load balancer there’s no automation to direct you to a healthy origin/service (if you were running multiple instances of your services). In other words, multiple tunnel connectors will help with availability if one of them was to go down, but it doesn’t help ensure traffic is routed to a healthy backend application.

A separate tunnel, with your current architecture, would be the quickest way with the least number of changes. Each tunnel connector establishes 4 connections back to Cloudflare’s edge, and since your services are single instance you wouldn’t be losing anything in terms of availability.

@waleed-alharthi

The second approach is more like what I’m after (so long as I don’t need to change the URL at the client side). I am using the same name for the docker network name on both machines but maybe they’re treated as 2 separate networks?

I use the exact same network name, service name and port on both servers hoping to map to a single URL. How’d I configure this page if the services had different ports? Or is there no escape from having different URLs or ports?

You wouldn’t need to change the client side URLs for this, it’d all be behind the scenes for end users. A Docker network name is just a label - there’s no connectivity between networks on multiple distinct hosts just because the name is the same.

I’m not sure if I follow your question? In the first example you had mentioned that your first server had three services (A, B and C), and the second server had three different services (D, E and F). Do you mean that you’re proposing to run A-F on both servers? If not, and your architecture is as described in your first post, you’d want to have a URL that’s accessible from either server to get to what you’ve described. You could do this in a few different ways - host file tricks, DNS that’s resolvable from both hosts, or just use the IP address of the server, all coupled with the port that the particular service is listening on.

If you can, I’d start with a new tunnel instead for your second host - it helps with separation of concerns, where each tunnel has access to a logically grouped set of services.

[waleed-alharthi]

Again, many thanks @colinbarr; looks like load balancers is what I’m after.

As for the services, my idea is to have services running on either servers (without replicas), and other services running with a replica (both servers).

Server 1: A, B, C (Primary)
Server 2: C (Replica), D, E