How to use EVP_DigestSignInit[_ex]()?

[Alessandro Vesely <vesely@xxxxxxx>]

Hi all,

my scarce crypto knowledge notwithstanding, I'm trying to maintain a DKIM 
signing function.  With the previous openssl version, I added a snippet to sign 
using ed25519, more or less like so:

    char *digest;              // digest already computed from
    size_t diglen;             // normalized message header

    EVP_PKEY *crypto_pkey;     // given

    size_t crypto_outlen;      // expected
    unsigned char *crypto_out;

    EVP_MD_CTX *md_ctx = EVP_MD_CTX_new();
    if (md_ctx == NULL) goto error;

    int status = EVP_DigestSignInit(md_ctx,
        NULL, NULL, NULL, crypto_pkey);
    if (status == 1)
        status = EVP_DigestSign(md_ctx,
            crypto_out, &crypto_outlen,
            digest, diglen);
    EVP_MD_CTX_free(md_ctx);
    if (status != 1) goto error;

That works.  With the onset of openssl 3 and RSA_sign() deprecation, I thought 
that snippet could work with RSA as well.  Given an RSA key, that code returns 
no error but a NULL crypto_out.  I should have specified SHA256 or (deprecated) 
SHA1, but I don't know how to obtain an EVP_MD *type.  Or is it better to use 
the _ex version?

Best
Ale
--