Hello everyone,
i am having some issues understanding the RSA_sign function:
RSA_sign(int type, const unsigned char *m, unsigned int m_len, unsigned char *sigret, unsigned int *siglen, RSA *rsa);
As far as I know, the signing is a four step process:
- Calculate hash with digest algorithm <type> (given as argument to this function -> m)
- Encapsulate hash in a DigestInfo structure (X509_SIG)
- Structure padding (in RSA_private_encrypt)
- Private key operation on this padded structure (in RSA_private_encrypt)
Is that correct?
So, during the TLS handshake, the RSA_sign function is called in the CertificateVerify step.
For my tests, everytime this function is called, the hashing type is SHA512 even though I specify to use a SHA256 hash.
These are the commands that I use to test TLS:
#openssl s_server -accept 443 -cert cert.pem -key key.pem -Verify 1 -msg -debug -cipher eNULL:aRSA:!SHA512:SHA256 -serverpref
#openssl s_client -connect localhost:443 -cert client_cert.pem -key client.key -state -cipher eNULL:aRSA:!SHA512:SHA256
How can I force TLS to use a SHA256 digest for DH?
