Can't issue certificate using certbot for server behind load balancer

Hey

I have a load balancer with 3 origins behind it, i am trying to get a certificate on each server for the DNS of the load balancer and i keep getting an error on my second server.
i have done the same process on the first server and it worked but as soon and i tried to issue a certificate using cerbot it failed.

this is the output

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer nginx

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: server-us-prod.wematch.live
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Running pre-hook command: service nginx stop
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for server-us-prod.wematch.live
Waiting for verification...
Cleaning up challenges
Running post-hook command: service nginx start
Failed authorization procedure. server-us-prod.wematch.live (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://server-us-prod.wematch.live/.well-known/acme-challenge/oc99rUS7ACgNTGjvvR82ftmURX3JKRJ1WBmluLwXExo: Connection refused

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: server-us-prod.wematch.live
   Type:   connection
   Detail: Fetching
   http://server-us-prod.wematch.live/.well-known/acme-challenge/oc99rUS7ACgNTGjvvR82ftmURX3JKRJ1WBmluLwXExo:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

Update 1:

i created this directory /.well-known/acme-challenge

now i get the following output

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Running pre-hook command: service nginx stop
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for server-us-prod.wematch.live
Waiting for verification...
Cleaning up challenges
Running post-hook command: service nginx start
Failed authorization procedure. server-us-prod.wematch.live (http-01): urn:ietf:params:acme:error:dns :: Fetching http://other-server.example.net/.well-known/acme-challenge/8ZpfS8juOcRycOahA2NHOBw0dJkmbUM0E3SwK5LHRyI: DNS problem: NXDOMAIN looking up A for other-server.example.net - check that a DNS record exists for this domain

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: server-us-prod.wematch.live
   Type:   None
   Detail: Fetching
   http://other-server.example.net/.well-known/acme-challenge/8ZpfS8juOcRycOahA2NHOBw0dJkmbUM0E3SwK5LHRyI:
   DNS problem: NXDOMAIN looking up A for other-server.example.net -
   check that a DNS record exists for this domain

i am using the nginx plugin for certbot and i have no idea why it just keeps on failing.
please help

Hi!

You won’t be able to use the HTTP-01 mechanism to request certificate as the inbound request will be randomly distributed to one of your three servers. There are a few options, in no particular order.

  1. Use a Cloudflare generated origin certificate. This only works if your traffic will all be going through Cloudflare as browsers do not recognize the signing authority as trusted, but Cloudflare will, so if you will be using Cloudflare in :orange: mode anyway, you won’t need to think about certificates again for years.

  2. Use DNS based validation methods instead. This will avoid the need to distribute the request to a specific origin server, although it is a slightly more complicated configuration and may rely on third party scripts.

  3. Copy the private key and public key between your servers yourself. How you distribute the certificates is up to you, but you could manage all certificates on one server and rsync, or use any other way of replicating the files.

  4. Configure your front end load balancers to deal with it. In this case, you would configure a 404 handler so that a request received by server #1 would be answered if the file exists, or reverse proxied to server #2 if not. Server #2 would do the same, forwarding to #3. #3 would forward back to #1. But, you’ll notice we have a loop here, and loops are bad, you will need to solve this yourself; this solution is a bit complex to begin with, so if you have concerns about loops, don’t even go down this route.

In my own case my load balancers obtain certificates using #2 (DNS), and distribute them using #3 (via a distributed file system). They also forward HTTP-01 validation requests to each backend server sequentially so that backend servers can still request certificates without storing the sensitive API keys needed to make DNS changes on the backend servers.

Fun? Maybe. Typed and edited from my iPhone, so don’t mind any typos, just blame autocorrect and assume it wasn’t me being dumb.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.