I'm having a problem connecting to particular machines via TLSv1.2
with the FIPS provider. The handshake fails with:
1022FDB6:error:0A000066:SSL routines:(unknown function):bad dh
value:ssl/statem/statem_clnt.c:2085:
and I can't figure out what the problem is. The weird thing is the
connection always succeeds with the default provider, but with the
FIPS provider it works with some servers (e.g. oracle.com) and fails
with other servers (e.g. debian.com). I have been able to reproduce
the problem with the openssl command line using options that force
the same cipher & key exchange that is negotiated by my
application code:
openssl s_client -4 -tls1_2 -sigalgs rsa_pkcs1_sha256 -cipher
DHE-RSA-AES128-GCM-SHA256 -trace -connect hostname:443
When using the FIPS provider and connecting to oracle.com this works
fine. The exact same command line fails with debian.com. Connections
to both machines work fine with the default provider. Both machines
use 4K RSA certificates.
What is causing OpenSSL FIPS to fail the DHE key exchange?
Incidentally, setting "security-checks = 0" in the configuration
file has no obvious effect on the problem.
Thanks,
Tom.III