On Sat, Apr 25, 2015 at 12:49:21AM +0000, Perrow, Graeme wrote:
> Using OpenSSL 1.0.1m on 64-bit Windows and Linux.
>
> I have implemented RSA encryption using the RSA_public_encrypt and
> RSA_private_decrypt functions and various padding types. This is working
> fine except that in very rare cases, my test fails because decrypting
> succeeds when it should fail. I'm basically doing this (pseudocode):
>
> RSA_public_encrypt( "abc", encrypted_data, my_public_key, RSA_PKCS1_OAEP_PADDING );
> RSA_private_decrypt( encrypted_data, decrypted_data, my_private_key, RSA_NO_PADDING );
A real code fragment would be substantially more useful that
"pseudo-code" here. This should *always* succeed, provided you
pass the correct length to RSA_private_decrypt. From the manpage:
flen must be ... and exactly RSA_size(rsa) for RSA_NO_PADDING.
> Note that the padding types are different. The vast majority of the time,
> I get an error from the RSA_private_decrypt call but now and again, it
> succeeds.
You're doing something wrong, it should always recover the OAEP
padded data, which is basically random, you need to reverse the
OAEP padding to recover the plaintext.
> I don't understand the underlying crypto well enough to know - is it
> possible for RSA_private_decrypt to be unable to tell that the wrong
> padding was used, or is this a bug in OpenSSL?
When not using padding, that's not "wrong" padding, you're just
doing a raw RSA decrypt, and any input that is smaller than the
modulus (but has the same bit length) should decrypt to something.
--
Viktor.
