Re: Handshake Failure : SSL_accept:Error in before SSL initialization

[Kamala Ayyar <kamala.ayyar@xxxxxxxxx>]

Hello Matt,

I call the WSAGetLastError() for Windows and that returns 183 (ERROR_ALREADY_EXISTS) //Cannot create a file when that file already exists

The SSL_get_error() gives us  SSL_ERROR_SYSCALL

Server code is roughly like below

SSL_CTX *m_pCtx;
SSL *m_pSsl;

m_pCtx = SSL_CTX_new(TLS_server_method();
if ((dwRet = LoadCertificates()) != rSUCCESS)
throw dwRet;
if ((m_pSsl = SSL_new(m_pCtx)) != NULL)
{
     if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket descriptor */
    {
    sslError = SSL_get_error(m_pSsl, iRet);
    LOGERROR(szLine);
    throw eSSL_ERROR;
    }
   SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
   ERR_clear_error();
   if ((sslError = SSL_accept(m_pSsl)) < 1)
     {
         sslError = SSL_get_error(m_pSsl, sslError);
         dwRet = handleError(sslError, "SSL_accept failed with error ", iRet);
         throw dwRet;// eSSL_ERROR;
     }
}

Client

SSL_CTX *m_pCtx;
SSL *m_pSsl;
m_pCtx = SSL_CTX_new(TLS_client_method();
if ((dwRet = LoadCertificates(TRUE)) != rSUCCESS) //Trust certificates only
throw dwRet;
/* Set for server verification*/
SSL_CTX_set_verify(m_pCtx, SSL_VERIFY_PEER, NULL); //Work in progress
m_pSsl = SSL_new(m_pCtx);
if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket descriptor */
{
   ssl_error = SSL_get_error(m_pSsl, iRet);
   LOGERROR(szLine);
   throw eSSL_ERROR;
}
SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
ERR_clear_error();
if ((iRet = SSL_connect(m_pSsl)) <= 0)   /* perform the connection */
{
ssl_error = SSL_get_error(m_pSsl, iRet);
dwRet = handleError(iRet, "SSL_connect failed with error ", ssl_error);
throw eSSL_ERROR;
}

ShowCerts();
}

As mentioned before this code works fine when  called by another application. So the certificates are all valid. I also tried this on different machines  but it did not work- I get the same error.

Thanks

Kamala

On Fri, Feb 4, 2022 at 12:20 PM Matt Caswell <matt@xxxxxxxxxxx> wrote:

Does errno give you anything?

How did you create your BIOs for m_pSsl?

Matt

On 04/02/2022 16:25, Kamala Ayyar wrote:
> Hello Matt,
>
> The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
> anything for this error, just an empty string.
> I use the following to print error but nothing is printed
> if ((retVal = SSL_accept(m_pSsl)) < 1)
> {
> sslError = SSL_get_error(m_pSsl, retVal);
> LOGERROR(getOpenSSLError());
> throw dwRet;// eSSL_ERROR;
> }
> string getOpenSSLError()
> {
> BIO *bio = BIO_new(BIO_s_mem());
> ERR_print_errors(bio);
> char *buf;
> size_t len = BIO_get_mem_data(bio, &buf);
> string ret(buf, len);
> BIO_free(bio);
> return ret;
> }
>
> *Kamala  Ayyar*
> 502 Claremont Ave.
> Teaneck NJ 07666-2563
> Tel: (201)530-0861
>
>
> On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt@xxxxxxxxxxx
> <mailto:matt@xxxxxxxxxxx>> wrote:
>
>
>
>     On 04/02/2022 15:17, Kamala Ayyar wrote:
>      >
>      > Hello,
>      >
>      > We are facing a strange handshake failure issue with a test
>     server and
>      > client application using OpenSSL in Windows.  We have tried with
>     both
>      > 1.1.1g and 3.0.1 versions- same problem. We created a Dll to
>     handle the
>      > OpenSSL functions- where the SSL context, SSL object and
>     certificates
>      > are handled. The certificates are obtained from the Windows store
>     and
>      > converted to cert and key using PKCS12_parse()
>      > The server accepts non secure connection from the client and then
>     passes
>      > the socket to the Dll that calls the TLS_server_method() and
>     creates the
>      > SSL context, SSL object and loads the certificates for use. It
>     however
>      > fails at SSL_accept(m_pSsl). We use a call
>      > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that
>     gave us
>      > the following error information
>      > SSL_accept:Error in before SSL initialization
>      > On the client side the same Dll is called with a client
>      > method TLS_client_method() and the error displayed
>     is SSL_connect:Error
>      > in SSLv3/TLS write client hello
>      > We have confirmed the certificates are good and valid.
>      >
>      > The same Dll called from a different heavily threaded application
>     with
>      > over 2000+ clients works well and handshake connections established
>      > without issues on a different port number.
>      >
>      > We have also tried to use OpenSSL methods directly without using
>     the Dll
>      > but we get the same failure.  This was also used with server and
>     client
>      > on the same machine as well as different machines with the same
>      > outcome.  The non secure communication works fine between the
>     server and
>      > the client
>
>     What does SSL_get_error() report after SSL_accept() fails?
>
>     Also please dump the OpenSSL error stack when it fails, e.g. using
>     something like ERR_print_errors_fp(stdout);
>
>     Matt
>