Do you mean RSA OAEP decryption done by HSM fails?
Possible tests:
1. Try RSA OAEP encryption/decryption with HSM. - basic test.
2. Encrypt with HSM and decrypt using openssl crypto library. - To make sure RSA OAEP encryption of the HSM works fine.
3. If test 2 fails, check if all the parameters (hash, mgf, salt length) used for OAEP are same on both sides. If they match and decryption still fails, check with your HSM vendor. If they don't, try fixing the parameters and repeat test 2.
RSA_NO_PADDING always works as all it does is modular exponentiation.
Thanks,
Thulasi.
On Mon, 17 Feb, 2020, 19:22 RudyAC, <rpo@xxxxxxxxxxxxxxx> wrote:
Hi,
I have the requirement to decrypt e-mails where RSA-OAEP padding is used. I
use the library openssl-1.0.2k and decrypt with CMS container (CMS_decrypt).
This works very well unless the private key is stored in a Hardware security
module and the cryptographic operation is performed via the PKCS11 engine
from openssl.
When decrypting an email which uses OAEP I got the error message:
47235129370352:error:06065064:digital envelope
routines:EVP_DecryptFinal_ex:bad decrypt:evp_enc.c:529:
To analyze the problem I encrypted an clear text using OAEP padding and
setup a decryption function using
RSA_private_decrypt(). Here I use padding mode "RSA_NO_PADDING" and the
decryption also works with the PKCS11 engine. Unfortunately CMS does not
support setting the padding mode.
For any comments I would be very grateful
Regards Rudy
--
Sent from: http://openssl.6102.n7.nabble.com/OpenSSL-User-f3.html
