On 02/07/2021 16:18, Reinier Torenbeek wrote:
It is not clear to me what you mean with "the OpenSSL engine interface does allow using EVP_PKEY_METHOD callbacks instead of rsa_priv_dec etc.". Can you elaborate (here or on the GitHub issue)?
You can hook the RSA calls at different abstraction levels. You can provide a custom RSA_METHOD in an enigne, which means calls to the various RSA_*() functions go via the custom RSA_METHOD.
However these RSA_*() functions are considered the "low level" functions, and it is preferred that applications use the "high level" EVP API instead (in fact the "low level" functions are all deprecated in 3.0). RSA PSS padding is *only* available via the EVP API.
Algorithm specific EVP functionality is implemented via an EVP_PKEY_METHOD. The built-in RSA EVP_PKEY_METHOD mostly just calls the low level RSA_*() functions in the right places. However it implements PSS padding directly. Since padding has already been added via the RSA_PKEY_METHOD, when it actually calls the lower level RSA_*() functions it does so with RSA_NO_PADDING. So this means that the RSA_METHOD has no opportunity to influence the PSS padding.
However, an alternative is to implement a custom EVP_PKEY_METHOD. By doing this you get the opportunity to hook the PSS padding.
Not sure I explained that too well. I hope it makes sense. Matt
