Hi,
I am working with a software vendor, and we're investigating what I currently believe is a bug in their software.
Their software is an HTTPS server. We provide the cert chain that
it uses to serve. Internally, it makes HTTPS requests to itself,
so in that regard, it also acts as a client, and will validate its
own certificate at that point. During that internal request, the
client currently fails to validate the server's certificate, and
that is the context of the problem we're investigating.
Currently, we supply a Let's Encrypt leaf cert & chain.
Currently, we provide Let's Encrypt's "default chain"; that is, we
take what LE gives us, and provide to this piece of software, so
that it can serve under the name on the cert. We are aware of the
alternate chain, but for simplicity's sake, I'd like to avoid it,
if possible. The default chain LE issues is leaf -> R3
intermediate -> ISRG root (cross signed by DST). The cert that
signs that last cert, the DST root, is now expired. However, the
IRSG root itself should be in any trust store.
As mentioned, when openssl s_client in this vendor's product attempts to verify the leaf (which is issues by LE), it fails:
Verify return code: 10 (certificate has expired)
While I know old versions of OpenSSL were affected by bugs that resulted in that error, the vendor here is not using a particularly old version:
# openssl version
OpenSSL 1.1.1l 24 Aug 2021
In fact, normally, this version of OpenSSL appears to correctly validate our leaf certificate. However, this vendor also does the following:
- they append our entire cert chain, leaf, intermediate and all, to /etc/ssl/certs/ca-certificates.crt
- they append an internal, self-signed CA to /etc/ssl/certs/ca-certificates.crt
If I remove either just that self-signed CA, or all 4 certs (my chain + the self signed) from ca-certificates.crt, then s_client shows that validation passes:
Verify return code: 0 (ok)
This is bizarre, to me, but in particular, that just removing an unrelated self-signed CA causes validation of a leaf not signed by that CA to succeed is really weird.
My point of view to the vendor has been that appending to
ca-certificates.crt is not valid; that instead, they should emit
root certs to something like /usr/local/share/ca-certificates
& run update-ca-certificates, and that the "interface" of this
directory includes also setting up the symlinks & hashed
symlinks appropriately for additional CAs. (And that it is my
understanding that update-ca-certificates handles this process for
oneself automatically.) That is, by merely appending to
ca-certificates.crt we're invoking some sort of undefined behavior
from the library. If this is documented somewhere, that'd be
helpful, I think, in convincing the vendor.
That said, the observed behavior is just weird, and I'd love a
deeper explanation of why removing a self-signed cert that seems
inconsequential from ca-certificates.crt appears to cause path
building to reach a different conclusion. Naïvely, removing a
trust anchor shouldn't cause more leaf certs to validate,
and that smells potentially like a bug in OpenSSL, excepting
potentially UB from the above (ab)use of ca-certificates.crt.
Thanks for any help,
—Roy
