Re: Handshake Failure : SSL_accept:Error in before SSL initialization

[Kamala Ayyar <kamala.ayyar@xxxxxxxxx>]

Hello Matt,

The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print anything for this error, just an empty string.

I use the following to print error but nothing is printed

if ((retVal = SSL_accept(m_pSsl)) < 1)
{
sslError = SSL_get_error(m_pSsl, retVal);
LOGERROR(getOpenSSLError());
throw dwRet;// eSSL_ERROR;
}

string getOpenSSLError()

{

BIO *bio = BIO_new(BIO_s_mem());
ERR_print_errors(bio);
char *buf;
size_t len = BIO_get_mem_data(bio, &buf);
string ret(buf, len);
BIO_free(bio);

return ret;

}

Kamala  Ayyar

502 Claremont Ave. 
Teaneck NJ 07666-2563 
Tel: (201)530-0861

On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt@xxxxxxxxxxx> wrote:

On 04/02/2022 15:17, Kamala Ayyar wrote:
>
> Hello,
>
> We are facing a strange handshake failure issue with a test server and
> client application using OpenSSL in Windows.  We have tried with both
> 1.1.1g and 3.0.1 versions- same problem. We created a Dll to handle the
> OpenSSL functions- where the SSL context, SSL object and certificates
> are handled. The certificates are obtained from the Windows store and
> converted to cert and key using PKCS12_parse()
> The server accepts non secure connection from the client and then passes
> the socket to the Dll that calls the TLS_server_method() and creates the
> SSL context, SSL object and loads the certificates for use. It however
> fails at SSL_accept(m_pSsl). We use a call
> back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that gave us
> the following error information
> SSL_accept:Error in before SSL initialization
> On the client side the same Dll is called with a client
> method TLS_client_method() and the error displayed is SSL_connect:Error
> in SSLv3/TLS write client hello
> We have confirmed the certificates are good and valid.
>
> The same Dll called from a different heavily threaded application with
> over 2000+ clients works well and handshake connections established
> without issues on a different port number.
>
> We have also tried to use OpenSSL methods directly without using the Dll
> but we get the same failure.  This was also used with server and client
> on the same machine as well as different machines with the same
> outcome.  The non secure communication works fine between the server and
> the client

What does SSL_get_error() report after SSL_accept() fails?

Also please dump the OpenSSL error stack when it fails, e.g. using
something like ERR_print_errors_fp(stdout);

Matt