Any help would be *much* appreciated. I am playing around with RSA signatures with different padding options and I have some questions.
I am trying to define different padding options and so am defining and using a EVP_PKEY_CTX . However I am not sure if this padding is getting used in the signature since my Verify outputs OK regardless of which option my Sign uses. Which leads to:
1 - Do I need to use a EVP_PKEY_CTX with the same options when doing verify? If so, I assume I can't reuse the same PKey_Ctx and I have to define another one. Right now even when I don't use any EVP_PKEY_CTX in Verify, I still verify OK, which makes me question if the padding option has been set.
2 - Is there a way to figure out what padding/hashing/etc option was used for the Sign/verify operation? This way I can be sure what algorithm or standard is being used.
3 - Do I need to set the hash function I am using in both EVP_PKEY_CTX as well as EVP_MD_CTX ? Or the latter is what defines this for the signing option?
4 - In general, is there a way of making the Signature/Encryptions in OpenSSL be deterministic for debugging/testing purposes?
5 - I noticed that there are two ways of determining the signature size: (a) by calling EVP_PKEY_size(rsaKeypair) as I am doing below, as well as (b) calling EVP_DigestSignFinal(md_ctx, nil, &sig_len) . Is one better than the other?
My sample code is below for reference. It's in Swift (but it should still be close enough to C to be readable). Also in Swift, some of the complex macros in OpenSSL have to be broken down to be compilable hence my usage of EVP_DigestUpdate instead of EVP_DigestVerifyUpdate .
let md_ctx = EVP_MD_CTX_create()
let md_ctx_verify = EVP_MD_CTX_create()
// To define padding option used in signature
let pkey_ctx = EVP_PKEY_CTX_new(rsaKeypair, nil)
// EVP_PKEY_CTX_set_rsa_padding(pkey_ctx, RSA_PKCS1_PADDING) - complex macro needs to be replaced
EVP_PKEY_CTX_ctrl(pkey_ctx, EVP_PKEY_RSA, -1, EVP_PKEY_CTRL_RSA_PADDING, RSA_X931_PADDING, nil)
// EVP_PKEY_CTX_set_signature_md() When should this be set?
// SIGN
var rc = EVP_DigestSignInit(md_ctx, &pkey_ctx, EVP_sha256(), nil, myRSA.rsaKeypair)
// EVP_DigestSignUpdate(md_ctx, message, message.count)
// Complex macro needs to be replaced
rc = EVP_DigestUpdate(md_ctx, message, message.count)
// allocate memory for signature
var sig_len: Int = Int(EVP_PKEY_size(rsaKeypair))
let sig = UnsafeMutablePointer<UInt8>.allocate(capacity: sig_len)
rc = EVP_DigestSignFinal(md_ctx, sig, &sig_len)
// VERIFY
rc = EVP_DigestVerifyInit(md_ctx_verify, nil, EVP_sha256(), nil, rsaKeypair)
// rc = EVP_DigestVerifyUpdate(md_ctx_verify, message, message.count)
rc = EVP_DigestUpdate(md_ctx_verify, message, message.count)
rc = EVP_DigestVerifyFinal(md_ctx_verify, sig, sig_len)
print("signature verified = \(rc == 1 ? "OK" : "FAIL")")
