FIPS requires that the DH parameters used in case of protocols that cannot transfer the q value (which TLS is) are known safe primes. Apparently the debian.com server does not use safe prime DH parameters with TLS-1.2. Nowadays it is better to just disable DHE ciphersuites with FIPS for maximum interoperability. Tomas Mraz, OpenSSL On Mon, 2023-08-14 at 18:37 -0700, Thomas Dwyer III wrote: > I'm having a problem connecting to particular machines via TLSv1.2 > with the FIPS provider. The handshake fails with: > > 1022FDB6:error:0A000066:SSL routines:(unknown function):bad dh > value:ssl/statem/statem_clnt.c:2085: > > and I can't figure out what the problem is. The weird thing is the > connection always succeeds with the default provider, but with the > FIPS provider it works with some servers (e.g. oracle.com) and fails > with other servers (e.g. debian.com). I have been able to reproduce > the problem with the openssl command line using options that force > the same cipher & key exchange that is negotiated by my application > code: > > openssl s_client -4 -tls1_2 -sigalgs rsa_pkcs1_sha256 -cipher DHE- > RSA-AES128-GCM-SHA256 -trace -connect hostname:443 > > When using the FIPS provider and connecting to oracle.com this works > fine. The exact same command line fails with debian.com. Connections > to both machines work fine with the default provider. Both machines > use 4K RSA certificates. > > What is causing OpenSSL FIPS to fail the DHE key exchange? > > Incidentally, setting "security-checks = 0" in the configuration > file has no obvious effect on the problem. > > > Thanks, > Tom.III > > -- Tomáš Mráz, OpenSSL
