How to plug in different digest algorithm implementation into the PKCS7 functions?

[stm@xxxxxxxxxx (Stephan Mühlstrasser)]

Am 26.04.16 um 16:25 schrieb Stephan M?hlstrasser:
> Hi,
>
> I'm trying to plug my own digest algorithm implementation into the PKCS7
> functions for creating a signature (using OpenSSL 1.0.2). The hash
> computation shall be performed on a hardware device.
>
> For that purpose I wanted to supply my own EVP_MD data structure to
> PKCS7_add_signature(). A rough sketch of my code for replacing the
> standard SHA-256 implementation looks like this:
>
>     static const EVP_MD my_digest_impl =
>     {
>         NID_sha256,
>         ...
>         /* contains function pointers for my own implementation */
>     };
>
>     PKCS7 *p7 = PKCS7_new();
>
>     PKCS7_set_type(p7, NID_pkcs7_signed);
>
>     PKCS7_SIGNER_INFO *si = PKCS7_add_signature(p7, cert, pkey,
> &my_digest_impl);
>
>     PKCS7_content_new(sig_parms->p7, NID_pkcs7_data);
>
>     PKCS7_set_detached(p7, 1);
>
>     BIO *p7bio = PKCS7_dataInit(p7, NULL);
>     ...
>
>...
> How can I plug in my own digest implementation? Do I need to implement a
> full OpenSSL engine for this purpose?

I was able to implement this requirement now by calling BIO_set_md() on 
the BIO that is created by PKCS7_dataInit(). The code for replacing the 
digest function looks like this (error checking omitted):

static const EVP_MD my_digest_impl =
{
      NID_sha256,
      ...
      /* contains function pointers for my own implementation */
};

EVP_MD_CTX *ctx;
BIO *p7bio = PKCS7_dataInit(p7, NULL);

BIO_get_md_ctx(p7bio, &ctx)

ctx->flags |= EVP_MD_CTX_FLAG_NO_INIT;

BIO_set_md(sig_parms->p7bio, &my_digest_impl);

ctx->update = my_digest_impl.update;
ctx->md_data = OPENSSL_malloc(my_digest_impl.ctx_size);

/* ... Now the ctx->md_data member is initialized with data specific to 
the hardware device ... */

my_digest_impl.init(ctx);

The use of the EVP_MD_CTX_FLAG_NO_INIT flag is necessary, because 
otherwise the digests init() function would be called from BIO_set_md() 
without the necessary information for initializing the hardware device. 
With the flag being set the data can be assigned to the md_data member 
after the call to BIO_set_md() and then the digest's init() function can 
be called.

I'd appreciate any comments if there's a problem with this approach. So 
far this seems to be working fine.

-- 
Stephan