Hello,
We are facing a strange handshake failure issue with a test server and client application using OpenSSL in Windows. We have tried with both 1.1.1g and 3.0.1 versions- same problem. We created a Dll to handle the OpenSSL functions- where the SSL context, SSL object and certificates are handled. The certificates are obtained from the Windows store and converted to cert and key using PKCS12_parse()
The server accepts non secure connection from the client and then passes the socket to the Dll that calls the TLS_server_method() and creates the SSL context, SSL object and loads the certificates for use. It however fails at SSL_accept(m_pSsl). We use a call back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that gave us the following error information
SSL_accept:Error in before SSL initialization
On the client side the same Dll is called with a client method TLS_client_method() and the error displayed is SSL_connect:Error in SSLv3/TLS write client hello
We have confirmed the certificates are good and valid.
The same Dll called from a different heavily threaded application with over 2000+ clients works well and handshake connections established without issues on a different port number.
We have also tried to use OpenSSL methods directly without using the Dll but we get the same failure. This was also used with server and client on the same machine as well as different machines with the same outcome. The non secure communication works fine between the server and the client
Thanks
Kamala
