526 error: ssl-cert points to defunct 'Google Trust Center LLC' despite cloudflare

I’ve attempted to create a ssl cert in cloudflare and point to it in my gke k8s deployment.
Despite verifying via kubectl that the certificate created is indeed the cloudflare cert, my domain (haiphen.io) still points to the defunct ‘Google Trust Domains’.

I’ve tried to DNS flush and redeploy the cluster several times. It seems to be some setting issue related to either cloudlfare / google domains that has locked in this certificate.

Just to check, you are not confusing your old GTS certificate with the Cloudflare edge certificate for your domain, that is also issued by GTS?
https://cf.sjr.org.uk/tools/check?3504bb01a4b648c1a5a7ed5de25b6271#connection-server-https

What certificate do you have installed on the origin itself?

I’m not sure how much info is appropriate to share on here.

I have a let’s encrypt certificate deployed as part of my gke k8s deployment. I’ve also created an origin server via cloudflare dashboard and passed the same keys to my gke cert-manager. Despite this, I still see the certificate affiliated with google domains:

$ openssl s_client -showcerts -servername haiphen.io -connect haiphen.io:443
Connecting to 2606:4700:3108::ac42:2af4
CONNECTED(00000005)
depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R1
verify return:1
depth=1 C=US, O=Google Trust Services LLC, CN=GTS CA 1P5
verify return:1
depth=0 CN=haiphen.io
verify return:1
---
Certificate chain
 0 s:CN=haiphen.io
   i:C=US, O=Google Trust Services LLC, CN=GTS CA 1P5
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 25 16:11:49 2024 GMT; NotAfter: Jul 24 16:11:48 2024 GMT

My dns server is hosted through google domains and it seems like something is overwriting the gke deployment. Lot’s more i can share

:point_up_2: This is your problem because you think…

The GTS certificate you see is the Cloudflare edge certificate, not your “old” certificate. See here…

If you want to see your own origin certificate, then set your DNS record to “DNS only” instead of “Proxied”, requests will then go direct to your origin. Note that as you will be bypassing Cloudflare, no Cloudflare protections or features can be applied to your site traffic until the site is proxied again.

1 Like

Indeed, for debugging, I’ve turned off the cloudflare proxy to connect directly through DNS. The certificate still points to a ‘google trust’ certificate whether the proxy is on or off. I’m not sure I will be able to entirely debug ‘my problem’ through the community chat alone. Are there any other cloudflare resources for real time chat?

Not for technical support for an issue that’s not due to Cloudflare.

With the proxy now off, I see your (expired) LE certificate on your origin…

* Server certificate:
*  subject: CN=haiphen.io
*  start date: Feb 29 06:38:55 2024 GMT
*  expire date: May 29 06:38:54 2024 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

haiphen.io then redirects to auth.haiphen.io which is proxied and so you see the Cloudflare edge GTS certificate again.

* Server certificate:
*  subject: CN=haiphen.io
*  start date: Apr 25 16:11:49 2024 GMT
*  expire date: Jul 24 16:11:48 2024 GMT
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1P5
*  SSL certificate verify ok.
2 Likes

re:

* Server certificate:
*  subject: CN=haiphen.io
*  start date: Feb 29 06:38:55 2024 GMT
*  expire date: May 29 06:38:54 2024 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

This is the blocker right here. The cert-manager on my gke deployment uses cloudflare api-key to generate the certificate:

# ./cert-manager/cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: [email protected]
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          email: [email protected]
          apiKeySecretRef:
            name: cloudflare-api-key-secret
            key: api-key

and when i deploy, I can see an updated certificate that should be live and active until Sept. 2024:

$ openssl x509 -in haiphen-io-tls.crt -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:96:69:89:ad:53:4c:c3:21:bc:03:76:9f:70:f1:38:ca:fc
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=R10
        Validity
            Not Before: Jun  7 14:57:33 2024 GMT
            Not After : Sep  5 14:57:32 2024 GMT
        Subject: CN=haiphen.io
...

It’s a bit of a grey area what/where things are breaking but that defunct certificate seems snuggled in there for good regardless of what I do to flush/replace it.

That’s the area I’m trying to address

Just so I’m clear, it isn’t a GTS issue, your issue is you have an expired LE certificate when you are running a script to renew it (using a DNS challenge set through the Cloudflare API)?

I don’t think it’s a Cloudflare issue then, just something you need to resolve on your origin to work out why you have the certificate but it’s not replacing the old one. I assume you’ve done the usual reload of the webserver.

Certificates are being issued as here…

2 Likes

My original message got overwritten by the png i tried attaching.

I can not rewrite everything but in summary i was saying the issue resolved after switching from full (strict) to full and the fact that cloudlfare gives a standard 526 error in this case is unhelpful at best for users needing to debug. I would suggest cloudlfare invest in more descriptive documentation in this area. I spent 2 week working to identify this. Cloudflare support was not effectively able to diagnose this either.

Now the question remains to identify why cloudflare servers cannot validate my origin server SSL certificate despite me seeing a valid certificate deployment on my side.

I see little room for confusion in the statement that an origin certificate is invalid. Testing is trivial using curl from any command line to check the certificate being sent by the origin server.

I’m not sure what detail Cloudflare could provide that would be any more clear. If you have some ideas that you think might improve the error message, you could always suggest them in Feedback.

@epic.network Your input is not at all helpful and I’m not interested in getting pulled into whatever maelstrom you’re baiting. I could equally label any network issue as solvable with some simple curl. Congrats on flexing your masterful knowledge of of google keywords. I offered the cloudflare team a perspective, you can chose to do with it whatever you will. Bullying clients out of meaningful engagement for the sake of quickly getting through tickets to hit metrics should not be mistaken for meaningful QA support. Move on.

I have no idea what you are insinuating with the “Google keywords” comment. If you want to me to show you how to easily use curl to test an origin certificate, I am more than happy to share that knowledge.

I am not sure why you think I am baiting anything or attempting to bully anyone. You seem to have taken my reply in a context that is substantially different that it was intended.

I am just a Cloudflare user like you. I have no metrics goals to meet.

You didn’t offer “the Cloudflare team” a perspective. You proposed it in a forum discussion with another Cloudflare user that may or may not be read by any Cloudflare staff. In the interest of getting your idea in front of a more suitable audience, I suggested that you propose your idea as a feature request. That was meant to encourage you to submit your idea with specific details of what you think would improve the error message!

If you decide to submit such a request, I would even happily vote for it if it could improve the user experience.

I hope your day gets better.

1 Like

Just for the record, you made your site insecure as you disabled proper encryption at this point and anyone could take over your requests without validation.

2 Likes

Because your testing is incorrect? No one in the forums has access to your origin server. If they did it is likely that their testing would indicate that the certificate served by your origin fails to meet the Full Strict requirements. Edit: I see @sjr demonstrated that to be true.

While it might be up for grabs as to whether or not @epic.network or I have debugged more of these scenarios I think I have an edge… it’s not ever in my experience been a failure on Cloudflare’s side to validate a legitimate origin certificate. That could change, but absent evidence Occam’s Razor applies.

1 Like

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.