OpenSSL 1.1.1b - TLS server handshake fails when using CAPI engine - capi_rsa_priv_enc() - capi engine: function not supported

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi,

This is regarding an issue reported here in this link - https://github.com/openssl/openssl/issues/8872  - This is blocking my development progress.
I'm seeing this issue - capi_rsa_priv_enc() - capi engine: function not supported, when I tried to upgrade my application from using OpenSSL version 1.0.2r to 1.1.1b.
This is causing TLS handshake to fail.
In my case, I'm creating a self signed certificate that is used by Server application loading CAPI engine.
Certificate is created by calling CertCreateSelfSignCertificate(--) - wincrypt.h - using szOID_RSA_SHA256RSA  signature algorithm and "Microsoft Enhanced RSA and AES Cryptographic Provider".

This failure doesn't look like it's because of TLS1.3 as turning off TLS1.3 while configure with "no-tls1_3" also caused the same problem.
Here is the call stack that is causing the reported problem when SSL_accept() is called.
 capi.dll!capi_rsa_priv_enc(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding)
libcrypto-1_1.dll!RSA_private_encrypt(int flen, const unsigned char * from, unsigned char * to, rsa_st * rsa, int padding)
libcrypto-1_1.dll!pkey_rsa_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libcrypto-1_1.dll!EVP_PKEY_sign(evp_pkey_ctx_st * ctx, unsigned char * sig, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libcrypto-1_1.dll!EVP_DigestSignFinal(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen)
libcrypto-1_1.dll!EVP_DigestSign(evp_md_ctx_st * ctx, unsigned char * sigret, unsigned int * siglen, const unsigned char * tbs, unsigned int tbslen)
libssl-1_1.dll!tls_construct_cert_verify(ssl_st * s, wpacket_st * pkt)
libssl-1_1.dll!write_state_machine(ssl_st * s)
libssl-1_1.dll!state_machine(ssl_st * s, int server)
libssl-1_1.dll!ossl_statem_accept(ssl_st * s)
libssl-1_1.dll!SSL_do_handshake(ssl_st * s)
libssl-1_1.dll!SSL_accept(ssl_st * s)


Is there a solution for this? Or
Do I need to switch to some other Signature algorithm like ECDSA? Can I use this or anything else instead of RSA?
If I should switch to ECDSA, should I use "szOID_ECDSA_SHA256" (wincrypt.h) as signature algorithm, use "PROV_EC_ECDSA_SIG" while calling CryptAcquireContext() and call ENGINE_set_default() with ENGINE_METHOD_EC to support ECDSA using capi.dll?

Thank you in advance.

--
Regards,
Sandeep


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux