Hi Matt,
Thank you for the inputs.
Yes, we had encountered the padding issue initially. But we added support for RSA_NO_PADDING in our hardware. That's why we are able to successfully decrypt the premaster secret in the hardware.
Hence the issue does not seem to be related to padding. We have confirmed this by comparing the premaster secret on both client and server and they are the same.
We suspect in this case, verification of "encrypted handshake message" failure is happening.
We understand constant_time_xx APIs get used for CBC padding validation. Will this have any dependency on the compiler optimization or asm flags? Will this issue be seen if hardware takes more time for the operation?
Here is the snippet of the wireshark where our device acting as server closes the connection with decryption failure.
If you need any further info, please let us know.
Thanks
Mahendra
Please suggest.
On Fri, Oct 30, 2020 at 3:32 PM Matt Caswell <matt@xxxxxxxxxxx> wrote:
On 30/10/2020 09:18, Mahendra SP wrote:
> Hi All.
>
> We have upgraded openssl version to 1.1.1b
>
> With this, we are seeing decryption error during SSL handshake for the
> below explained scenario. Our device acts as an SSL server.
>
> We have external hardware to offload RSA private key operations using
> the engine.
> Decryption of pre-master secret is done using hardware and is
> successful. We compared the pre-master secret on both server and client
> and they match.
> However, we see that SSL handshake fails with "decrypt error (51)" with
> an alert number 21. Verifying the encrypted finish message on the server
> side fails.
>
> This issue does not happen with software performing RSA private key
> operations.
>
> Can someone help with the reason for decryption failure? Below is the
> compiler and processor details. It is 64 bit.
> arm-linux-gnueabihf-gcc -march=armv7ve -mthumb -mfpu=neon -mfloat-abi=hard
Potentially this is related to the use of PSS padding in libssl which is
mandated in TLSv1.3. The TLSv1.3 spec also requires its use even in TLSv1.2.
The PSS padding is implemented within the EVP layer. Ultimately EVP
calls the function RSA_private_encrypt() with padding set to RSA_NO_PADDING.
Assuming your engine is implemented via a custom RSA_METHOD does it
support RSA_private_encrypt(() with RSA_NO_PADDING? If not this is
likely to be the problem.
More discussion of this is here:
https://github.com/openssl/openssl/issues/7968
Also related is the recent discussion on this list about the CAPI engine
and this issue:
https://github.com/openssl/openssl/issues/8872
Matt
