Unable to Verify Certificate as of about 5 am EST

10.70 Could not fetch URL https://pypi.org/simple/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host=‘pypi.org’, port=443): Max retries exceeded with url: /simple/pip/ (Caused by SSLError(SSLCertVerificationError(1, ‘[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1135)’))) - skipping
certificate (_ssl.c:1135)‘))’: /simple/nltk/

This just started on me this morning. It looks like there is reference to this, but also not really mentioning North America (NE cooridor). Is there any resolution in sight? Or is there a new key or something that I need with the recently reported change to Google Trust? Or am I the only having this problem in this fashion? Thanks in advance.

Am I missing some context here or what does this have to do with Cloudflare? pypi uses Fastly and it looks like that error is from pip.

1 Like

It is part of the middleware application yes. However, while working all along, it suddenly can’t verify the certificates, which are handled here. And my website is down, which I have now as my company for my security certificates. I read that the change to Google Trust had over a million sites down, but I don’t know if that’s it. If I need to generate a new certificate, and go with that, now that this says “Google Trust” on my account / security certificate, then I will do that, but this has been working fine for 6 months, and has now gone no when trying to check the certificate. Mine says it’s good until 2039, and there is still a backup that has never been deployed. Don’t get me to lying though. I am very new to this, and I am just trying to find out why it stopped working. Going to the Website now just gives me a 502 error. Same works on my localhost which is what is connecting to my users. So, I am not blaming anyone, but wondering if I need to shake another tree. Sorry if that is confusing, but all I know is that I found out the site was down. Tried to do a rebuild, and then suddenly was seeing the error that it couldn’t validate the SSL certificate. I am an open book. Please let me know if there is somewhere else to hunt, but since Cloudflare is where I have elected to have my SSL Certificates, I thought this made the most sense to start. Literally happened over night. (Apparently there is mention of an outage that lasted over 5 hours when transitioning from JDNC? to Google Trust, and that article came out this morning, dated 05/02/24 speaking to millions of affected customers. So, hopefully someone will be able to verify, or point me in the right direction. Thanks for the response Chaika!

This is the article that I found here:

jsDelivr May outage postmortem
During the night, on May 2, 2024, the jsDelivr CDN domain cdn.jsdelivr.net started serving an expired SSL certificate to clients connecting from certain regions.

The outage lasted for more than 5 hours and affected users mostly in Africa, Asia, and certain countries in Europe and Latin America.

Users from the USA, Canada, western Europe, Brazil, and many other countries were not affected.

This disparity was due to our routing between our main CDN providers. Users that were hitting our Fastly CDN endpoint were unaffected.

The root cause of the outage was Cloudflare’s switch from DigiCert’s certificate authority to Google Trust Services. While the switch itself was benign, it also changed the domain validation method.

Since we’re a multi-CDN, with the traffic being routed between providers based on our own internal rules, we can’t use Cloudflare DNS hosting and have a special setup where they only act as the CDN, with DNS being hosted elsewhere.

To allow Cloudflare to automatically issue and manage our certificates, we added the proper DNS records to our third-party DNS providers. This system worked great for almost 10 years now.

Unfortunately, this migration of certificate authorities also made those validation records obsolete, and switched to HTTP validation instead. This would never work in our case, as depending on where the verification test came from, it could hit a different CDN provider and fail.

We were not aware of this change, so what ended up happening was the following:

The previous working DigiCert was set to expire, as it has happened many times over the years
Some time shortly before the expiration, Cloudflare tried to issue the new certificate using HTTP validation and failed
After it failed, it reverted to an old certificate expired in 2020
None of us or our systems ever expected the managed SSL system to fail
This resulted in an error message to all of our users hitting Cloudflare’s CDN based on our routing
I want to note that Cloudflare is at no fault here, and they have no obligation towards anyone to validate rare client configs, especially free sponsored ones.

I, @jimaek, take full responsibility for this outage. I should’ve not made any assumptions about the migration and should’ve taken extra precautions.

This outage is embarrassing, and I apologize to everyone who was affected!

The goal of jsDelivr was always to be a production-ready, most reliable, and fastest open-source CDN for anyone to use. This never changed and never will.

The next steps are:

Short-term:
Review the full system to ensure that similar issues are automatically handled and re-routed
From now on, any critical changes by CDN providers will immediately result in their deactivation from jsDelivr and manual verification after the fact to ensure the CDN’s stability with our specific setup
Long-term:
Optimize, automate, and simplify our DNS, load-balancing, and failover systems and components to prevent other theoretically possible but unknown edge cases
We will integrate our own Globalping service for even better and more reliable monitoring and failover. Consider joining the project by hosting a Docker probe on your network!
Thank you to everyone for being with us for so many years, and hope to see you all continue this journey forward.