Hi,
This is regarding an issue reported here in this link - https://github.com/openssl/openssl/issues/8872 - This is blocking my development progress.
I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not supported, when I tried to upgrade my application from using OpenSSL version 1.0.2r to 1.1.1b.
This is causing TLS handshake to fail.
In my case, I'm creating a self signed certificate that is used by Server application loading CAPI engine.
Certificate is created by calling CertCreateSelfSignCertificate(--) - wincrypt.h - using szOID_RSA_SHA256RSA signature algorithm and "Microsoft Enhanced RSA and AES Cryptographic Provider".
This failure doesn't look like it's because of TLS1.3 as turning off TLS1.3 while configure with "no-tls1_3" also caused the same problem.
Here is the call stack that is causing the reported problem when SSL_accept() is called.
capi.dll!capi_rsa_priv_enc(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding)libcrypto-1_1.dll!RSA_private_encrypt(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding)libcrypto-1_1.dll!pkey_rsa_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)libcrypto-1_1.dll!EVP_PKEY_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)libcrypto-1_1.dll!EVP_DigestSignFinal(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen)libcrypto-1_1.dll!EVP_DigestSign(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)libssl-1_1.dll!tls_construct_cert_verify(ssl_st * s, wpacket_st * pkt)libssl-1_1.dll!write_state_machine(ssl_st * s)libssl-1_1.dll!state_machine(ssl_st * s, int server)libssl-1_1.dll!ossl_statem_accept(ssl_st * s)libssl-1_1.dll!SSL_do_handshake(ssl_st * s)libssl-1_1.dll!SSL_accept(ssl_st * s)
Is there a solution for this? Or
Do I need to switch to some other Signature algorithm like ECDSA? Can I use this or anything else instead of RSA?
If I should switch to ECDSA, should I use "szOID_ECDSA_SHA256" (wincrypt.h) as signature algorithm, use "PROV_EC_ECDSA_SIG" while calling CryptAcquireContext() and call ENGINE_set_default() with ENGINE_METHOD_EC to support ECDSA using capi.dll?
Thank you in advance.
Regards,
Sandeep
Sandeep
