Authenticated Origin Pulls - Handshake failed

Hi Cloudflare community!

I’m setting up a new Ghost instance and after the installation went very well but trying to enable Authenticated Origin Pulls is being a pain since yesterday. I believe I created and detroyed 5 hosts trying to make this work.

Context. I tried this guides:

but whenever I enable “Authenticated Origin Pulls” on CloudFlare I receive the following error:

** Error 525 - SSL handshake failed **

The Cloudflare certificate (origin-pull-ca.pem) is imported as /etc/ssl/certs/cloudflare.crt

And on nginx domain-ssl.conf I inserted:
ssl_client_certificate /etc/ssl/certs/cloudflare.crt;
ssl_verify_client on;

FWIW: I didn’t removed the ssel_certificate and ssl_certificate_key (LetsEncrypt).
FWIW²: I’m playing with two domains around this issue. On one of this domains I generated a Origin certificate but no luck as well.

Technical:
Host: DigitalOcean Droplet
System: Ubuntu 20.04
Website: https://joelteixeira.com

Could you please shed some light here?

Att,
Joel Teixeira

I have to admit that I haven’t tested it with an SSL certificate other that Cloudflare Origin CA certificate being generated at Cloudflare dashboard and then installed at my host/origin, while using Authenticated Origin Pulls.

What Nginx version are you running and you are running on Ubuntu right?

What is the response if you execute the below command (change your origin IP with your real one)?:
curl -v --resolve joelteixeira.com:443:[your origin IP] https://joelteixeira.com

Also, have you got ca-certificates installed?
Which openssl version are you running ( openssl version -a)?

Is the SSL certificate a valid one and not being expired?

Are both of the A www and A yourdomain.com (or CNAME?) being :orange: cloud?

My vhost file for mydomain looks like this below:

  1. Allowed Cloudflare IP addresses to connect to my host/origin.
  2. Checked and enabled SSL and 443 port at my host/origin.
  3. Make sure it rewrites from HTTP to HTTP at host/origin too (while keeping non-www to www redirection).
  4. Having generated an Cloudflare Origin CA certificate and installed it at my host/origin.
  5. Enabled Authenticated Origin Pulls option and added an Authenticated Origin Pulls certificate too.
  6. Enabled Full SSL (Strict) at Cloudflare dashboard.
  7. Having both Always Use HTTPS and Automatic HTTPS Redirection enabled.
  8. HSTS option at Cloudflare is also enabled.
  9. Minimum TLS 1.2 at Cloudflare selected.
  10. Other stuff: DNSSEC is enabled for my domain.
server {
    listen *:80;
    listen [::]:80;
    listen *:443 ssl http2;
    listen [::]:443 ssl http2;

    ssl_protocols TLSv1.2 TLSv1.3;

    # Cloudflare Origin CA certificate for mydomain.com, www.mydomain.com
    ssl_certificate /var/www/clients/client1/web1/ssl/mydomain.com.crt;
    ssl_certificate_key /var/www/clients/client1/web1/ssl/mydomain.com.key;

    # Cloudflare Authenticated Origin Pulls
    ssl_client_certificate /etc/origin-pull-ca.pem;
    ssl_verify_client on;

    ssl_session_timeout 1d;
    ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
    ssl_session_tickets off;

    # curl https://ssl-config.mozilla.org/ffdhe2048.txt > /path/to/dhparam
    ssl_dhparam /etc/dhparam-mozilla.pem;

    # intermediate configuration
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    server_name mydomain.com www.mydomain.com;

    root   /var/www/mydomain.com/web/;
    disable_symlinks if_not_owner from=$document_root;

    if ($scheme != "https") {
        rewrite ^ https://$http_host$request_uri? permanent;
    }
    if ($http_host = "mydomain.com") {
        rewrite ^ $scheme://www.mydomain.com$request_uri? permanent;
    }

Once Authenticated Origin Pulls are enforced by your origin server, any HTTPS requests outside of Cloudflare to your origin will fail including those to gray clouded records on Cloudflare.

May I ask if you re-check if the Cloudflare is allowed to connect to your DigitalOcean droplet?

Regarding 525 error, have you tried checking with the steps from the below article?:

1 Like

Wow, that’s a lot of substantial information. I didn’t ran thought the “allowing cloudflare ip addresses”. Maybe that’s the catch.

The nginx version is:
nginx version: nginx/1.18.0 (Ubuntu)

I’ll update this post after more tests based on your valuable input.

Thank you @fritexvz

1 Like

@fritexvz

Results for “curl -v --resolve joelteixeira.com:443:[your origin IP] https://joelteixeira.com”.

* Expire in 0 ms for 6 (transfer 0x7928b0)
* Added joelteixeira.com:443:[MY_IP] to DNS cache
* Hostname joelteixeira.com was found in DNS cache
*   Trying [MY_IP]...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x7928b0)
* connect to [MY_IP] port 443 failed: Connection refused
* Failed to connect to joelteixeira.com port 443: Connection refused
* Closing connection 0
curl: (7) Failed to connect to joelteixeira.com port 443: Connection refused

ca-certificates:

ca-certificates is already the newest version (20210119~20.04.1).

openssl
OpenSSL 1.1.1f 31 Mar 2020
built on: Mon Mar 22 11:37:17 2021 UTC
platform: debian-amd64
options: bn(64,64) rc4(16x,int) des(int) blowfish(ptr)
compiler: gcc -fPIC -pthread -m64 -Wa,–noexecstack -Wall -Wa,–noexecstack -g -O2 -fdebug-prefix-map=/build/openssl-Juj39H/openssl-1.1.1f=. -fstack-protector-strong -Wformat -Werror=format-security -DOPENSSL_TLS_SECURITY_LEVEL=2 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG -Wdate-time -D_FORTIFY_SOURCE=2
OPENSSLDIR: “/usr/lib/ssl”
ENGINESDIR: “/usr/lib/x86_64-linux-gnu/engines-1.1”
Seeding source: os-specific

About the DNS configuration:

I whitelisted all cloudflare IPs:
iptables -I INPUT -p tcp -m multiport --dports http,https -s 173.245.48.0/20 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 103.21.244.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 103.22.200.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 103.31.4.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 141.101.64.0/18 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 108.162.192.0/18 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 190.93.240.0/20 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 188.114.96.0/20 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 197.234.240.0/22 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 198.41.128.0/17 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 162.158.0.0/15 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 104.16.0.0/12 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 172.64.0.0/13 -j ACCEPT
iptables -I INPUT -p tcp -m multiport --dports http,https -s 131.0.72.0/22 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2400:cb00::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2606:4700::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2803:f800::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2405:b500::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2405:8100::/32 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2a06:98c0::/29 -j ACCEPT
ip6tables -I INPUT -p tcp -m multiport --dports http,https -s 2c0f:f248::/32 -j ACCEPT

About the vhost, it’s almost the same. I believe the only difference is the generated Origin certificate. I’ll give it a try again. I shound just comment the certificate lines for letsencrypt right?

Thank you.

Just to update, I tried again to generate a Origin Certificate and imported to the server.

If turn off the “Origin Pulls” it gives me:
image

If I turn it on:

If I comment out the Origin Pulls lines:
image

And turn off the Origin Pulls it opens the site:

Certificate:

Have you run iptables-save at the end to apply changes?

That is normal and expected response.

That seems kind of strange.

Interesting.

1 Like

oh! My bad. I didn’t.

Executed now:

IPV4
image

IPV6
image

But no changes AFAICT.

Not sure if is expected but when using the Origin Certificate I have this warning (doesn’t happen with Let’s Encrypt).

nginx -t

As it is just warning, but have you got ssl_stapling on; or ssl_stapling off; (also the ssl_stapling_verify on or off)in your Nginx vhost config file?

I do not have that warning.

Is your website working now?

If needed, you can re-check your nginx vhost config file using this example here:

1 Like

That’s weird, I don’t seem to have this setting anywhere on my files (domain conf files). And if I just comment out the origin certificates and use the letsencrypt the warning message does not happen.

image

Attaching the way the files are now:
…/sites-enabled/default

server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
try_files $uri $uri/ =404;
}
}

…/sites-enabled/joelteixeira.com.conf

server {
listen 80;
listen [::]:80;
server_name joelteixeira.com;
root /var/www/ghost/system/nginx-root; # Used for acme.sh SSL verification (https://acme.sh)
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:2368;
}
location ~ /.well-known {
allow all;
}
client_max_body_size 50m;
}

…/sites-enabled/joelteixeira.com-ssl.conf

server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name joelteixeira.com;
root /var/www/ghost/system/nginx-root; # Used for acme.sh SSL verification (https://acme.sh)
ssl_certificate /etc/letsencrypt/joelteixeira.com/fullchain.cer;
ssl_certificate_key /etc/letsencrypt/joelteixeira.com/joelteixeira.com.key;
ssl_client_certificate /etc/ssl/certs/cloudflare.crt;
ssl_verify_client on;
include /etc/nginx/snippets/ssl-params.conf;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_pass http://127.0.0.1:2368;
}
location ~ /.well-known {
allow all;
}
client_max_body_size 50m;
}

For some reason, only happens with the origin certificate.

It’s not, I’m keeping the Origin setting ON for now.
It works if I turn off together with the Origin Pulls and remove the ssl_client_certificate and ssl_verify_client lines from joelteixeira.com-ssl.conf.

I did not knew this mozilla tool, nice tip. Thank you again @fritexvz. I’ll play a little with it.

FWIW: The only modification after the deploy of this server was the ghost install process (and its dependencies). I’ll try to start a new one and set up SSL before installing ghost.

Some more info, following one of the guides you linked:

Using letsencrypt:

Using origin certificate:

There are 2 errors actually, 403 and 525
Error 403: Forbidden
If you’re seeing a 403 error without Cloudflare branding, this is always returned directly from the origin web server, not Cloudflare, and is generally related to permission rules on your server. The top reasons for this error are: 1. Permission rules you have set or an error in the .htaccess rules you have set 2. Mod_security rules. 3. IP Deny rules Since Cloudflare can not access your server directly, please contact your hosting provider for assistance with resolving 403 errors and fixing rules. You should make sure that Cloudflare’s IPs aren’t being blocked.

Cloudflare will serve 403 responses if the request violated either a default WAF rule enabled for all orange-clouded Cloudflare domains or a WAF rule enabled for that particular zone. Read more at What does the Web Application Firewall do? Cloudflare will also serve a 403 Forbidden response for SSL connections to sub/domains that aren’t covered by any Cloudflare or uploaded SSL certificate.

If you’re seeing a 403 response that contains Cloudflare branding in the response body, this is the HTTP response code returned along with many of our security features:

  • Web Application Firewall challenge and block pages
  • Basic Protection level challenges
  • Most 1xxx Cloudflare error codes
  • The Browser Integrity Check
  • If you’re attempting to access a second level of subdomains (eg- *.*.example.com ) through Cloudflare using the Cloudflare-issued certificate, a HTTP 403 error will be seen in the browser as these host names are not present on the certificate.

If you have questions contact Cloudflare Support and include a screenshot of the message you see or copy all the text on the page into a support ticket.
Error 525: SSL handshake failed
525 errors are often caused by a configuration issue on the origin web server. Error 525 occurs when these two conditions are true:

  1. The SSL handshake fails between Cloudflare and the origin web server, and
  2. Full or Full (Strict) SSL is set in the Overview tab of your Cloudflare SSL/TLS app.

Resolution

Contact your hosting provider to exclude the following common causes at your origin web server:

  • No valid SSL certificate installed
  • Port 443 (or other custom secure port) is not open
  • No SNI support
  • The cipher suites accepted by Cloudflare does not match the cipher suites supported by the origin web server

If 525 errors occur intermittently, review the origin web server error logs to determine the cause. Configure Apache to log mod_ssl errors. Also, nginx includes SSL errors in its standard error log, but may possibly require an increased log level.

1 Like

Found it. It’s present on this file:
/etc/nginx/snippets/ssl-params.conf

that is referencied on my domain-ssl.conf:
include /etc/nginx/snippets/ssl-params.conf;

It seems is something related to this, I’m still troubleshooting here but when I deleted all my domain.conf files and rebuilt it using the configuration that you kindly inserted on your first answer it worked. Trying parameter by parameter here to identify exatcly what.

@william.haskin25 thank you for your input as well. I don’t know deeply this configurations and the bullets that you and @fritexvz are pointing it’s extremally helpful.

1 Like

While I don’t have any problems with CF Authenticated Origin Pulls myself, without knowing your setup, maybe better to ditch it and use a better alternative on offer from Cloudflare now - Argo Tunnels via CF Teams free subscription does what Authenticated Origin Pulls does too protect non-CF IP access to site once you configure your origin server firewall to block all non-CF IP traffic. I wrote a guide for Argo Tunnels at https://blog.centminmod.com/2021/02/09/2250/how-to-setup-cloudflare-argo-tunnel-on-centos-7/ for my users

1 Like

you can also try troubleshooting by changing temporarily to ssl_verify_client optional; Module ngx_http_ssl_module and restart nginx and retest via curl verbose request to your real server IP and see the header output

1 Like
  1. Do you run your Ghost instance on the whole host/origin server or you have more than one domain at your host/origin?
  2. Do you use a yourdomain.vhost config file for your domain or modifying the general nginx.conf file?
  3. proxy_pass and X-Forwarded-For, X-Real-IP …

Okay, forget 1) and 2), I see you have default and joelteixeira.com.conf and joelteixeira.com-ssl.conf.

Would be good to have a single, which can handle both HTTP and HTTPS for you, because now I currently do not know if you change anything and in which one, which is actually efective to work so.

Regarding point 3) have you done this?:

Inside your Nginx.conf file and inside http {...} block, you should have:

set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 104.16.0.0/12;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2c0f:f248::/32;
set_real_ip_from 2a06:98c0::/29;

real_ip_header CF-Connecting-IP;
# real_ip_header X-Forwarded-For;

You are proxying your Ghost app over a HTTP on a port and the outgoing port is 443, right?

I am a bit afraid about this one as it could be out of scope of this forums regarding the Nginx configuration at your host/origin.
But, at first, and in few tires as far as you written, it actually works as you have it so close done within the stated as needed.
If we continue to reply I am not sure if you would manage to get it working without any assistance from someone else, because I do not see a Cloudflare issue here and seems to me it has nothing to do with Cloudflare so far.

I see you understand what you need to do, and you have setup it kind a good, just there are some misconfigurations in between.

I am sorry, but I have not yet used Ghost setup on Nginx over proxy_pass, and having a separate config for HTTP and HTTPS, so I really cannot provide any other better suggestion.

Kindly, may I ask if you can try adding some other domain and try to setup Cloudflare using Cloudflare Origin CA Certificate and enable the Authenticated origin Pulls for that domain and see how that works out and reply back here?

I mean not using Ghost, just a simple php file like “Hello world” if you can get it working with above stated?

1 Like

I don’t have this code inside my file. You are right. The best from beginning would be set a minimal enviroment just to validate this setting. After the deploy I just installed the dependencies and run the ghost installer. Looks like the way it deploy the webserver is not the ideal indeed.

Interesting finding:
1 - It’s working now even without the Origin certificate. So in fact the Origin Pulls can work with the LetsEncrypt certificate that is auto generated during Ghost install.

2 - Better late than never. With all the help and support from you guys I finally know what was causing the handshake error. I did my best to restore everything it was exactly on the beginning and changing one single line it worked:

Inside joelteixeira.com-ssl.conf I have this line:
include /etc/nginx/snippets/ssl-params.conf;

and inside /etc/nginx/snippets/ssl-params.conf
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

changing it to:
ssl_protocols TLSv1.2 TLSv1.3;

And it works.

Thank you all guys!

1 Like

Big :clap: and a great go with minimal and your effort!

Thank you for usefull feedback information for future!

Yes, that, as from my post was TLSv1.2 TLSv1.3. And the fact I did not know what else could you possible have in your ssl-params.conf added which, obviously resulted in a misconfiguration.

I do not say it is bad to have it splitted in few config files, but it can be kind of a difficult to manage, either having it “all in one”, but at least on my place knowing what it is :slight_smile:

I happy and glad you found a solution :wink:

1 Like

Exactly fritexvz. When I rebuilt a new configuration based on yours and worked I started test every setting to understand which exatcly was affecting. In a next opportunity I will be more careful to reference all related files. Thank you man.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.