Signing using EVP_PKEY_encrypt when using pkcs11 engine

[Martin Townsend <mtownsend1973@xxxxxxxxx>]

Hi,

I'm trying to modify the evm/ima utility so that it can use a HSM to
perform signing.  I've setup SoftHSM and used this to create a
certificate with an RSA public key pair.  The evmctl code creates the
hash and then calls a function to perform the sign operation which
ends up calling
len = RSA_private_encrypt(size + asn1->size, buf, hdr->sig,
  key, RSA_PKCS1_PADDING);

My idea was to keep the hash calculation as is, and replace the
RSA_private_encrypt with code that uses the private key in the HSM to
encrypt the hash buffer that has been calculated.

My initialisation looks like this
    /* Load the configuration using OPENSSL_CONF environment variable */
    OPENSSL_config(NULL);
    /* Try and load PKCS11 engine */
    const char* s = getenv("OPENSSL_CONF");
    printf("Trying to load pkcs#11 engine\n");
    printf("OPENSSL_CONF=%s\n", s);
    pkcs_engine = ENGINE_by_id("pkcs11");
    if (!pkcs_engine) {
        printf("PKCS#11 engine not found, not using HSM\n");
    } else {
        int rv = ENGINE_init(pkcs_engine);

        if (!rv) {
            fprintf(stderr, "PKCS#11 could not be initialised\n");
            ENGINE_free(pkcs_engine);
            pkcs_engine = NULL;
        }

        ENGINE_set_default(pkcs_engine, ENGINE_METHOD_ALL);
    }

    OpenSSL_add_all_algorithms();
    OpenSSL_add_all_digests();
    ERR_load_crypto_strings();

and then I load the private key with

    key = ENGINE_load_private_key(pkcs_engine, keyid, UI_OpenSSL(), NULL);
    if (!key) {
        log_err("%s: Failed to load private key with id: %s\n", keyid,
__func__);
        ERR_print_errors_fp(stderr);
    }


and then use the following to perform the encryption

        /* Create context */
        hsm_key_ctx = EVP_PKEY_CTX_new(hsm_key, NULL);
        if (hsm_key_ctx == NULL) {
            log_err("sign_hash_v2: failed to create context\n");
            ERR_print_errors_fp(stderr);
            return -1;
        }
        rv = EVP_PKEY_encrypt_init(hsm_key_ctx);
        if (rv <= 0) {
            log_err("sign_hash_v2: failed to init encrypt (rv=%d\n", rv);
            ERR_print_errors_fp(stderr);
            EVP_PKEY_CTX_free(hsm_key_ctx);
            return -1;
        }
        /* TODO: What padding??? RSA_PKCS1_PSS_PADDING?? */
        rv = EVP_PKEY_CTX_set_rsa_padding(hsm_key_ctx, RSA_PKCS1_PADDING);
        if (rv <= 0) {
            log_err("sign_hash_v2: failed to set RSA_PKCS1_PADDING
(rv=%d\n", rv);
            ERR_print_errors_fp(stderr);
            EVP_PKEY_CTX_free(hsm_key_ctx);
            return -1;
        }
        if (rv <= 0) {
            log_err("sign_hash_v2: failed to set RSA_PKCS1_PADDING
(rv=%d\n", rv);
            ERR_print_errors_fp(stderr);
            EVP_PKEY_CTX_free(hsm_key_ctx);
            return -1;
        }

        /* Create signature */
        outlen = 0;
        rv = EVP_PKEY_encrypt(hsm_key_ctx, NULL, &outlen, buf, size +
asn1->size);
        if ((rv <= 0) || (outlen == 0)) {
            log_err("sign_hash_v2: failed to learn needed output buf
len (rv=%d)\n", rv);
            ERR_print_errors_fp(stderr);
            EVP_PKEY_CTX_free(hsm_key_ctx);
            return -1;
        }
        /*outlen = 256;*/
        log_info("EVP_PKEY_encrypt: outlen: %lu\n", outlen);
        rv = EVP_PKEY_encrypt(hsm_key_ctx, hdr->sig, &outlen, buf,
size + asn1->size);
        if (rv <= 0) {
            log_err("sign_hash_v2: EVP_PKEY_encrypt() failed (rv=%d)\n", rv);
            ERR_print_errors_fp(stderr);
            EVP_PKEY_CTX_free(hsm_key_ctx);
            return -1;
        }

But I find that when I create a signature for a particular file it's
always different with each invocation of the utility. If I try and
verify it using the associated public key I get:

RSA_public_decrypt() failed: -1
errno: No data available (61)
error:0407006A:rsa routines:RSA_padding_check_PKCS1_type_1:block type is not 01
error:04067072:rsa routines:RSA_EAY_PUBLIC_DECRYPT:padding check failed

Is there something I am missing in my code above?  I tried setting the
Engine in EVP_PKEY_CTX_new but get:
sign_hash_v2: failed to create context
140174165591744:error:260C0065:engine
routines:ENGINE_get_pkey_meth:unimplemented public key
method:tb_pkmeth.c:128:
140174165591744:error:0609D09C:digital envelope
routines:INT_CTX_NEW:unsupported algorithm:pmeth_lib.c:166:
errno: Invalid argument (22)

Any help appreciated,
Martin.