Re: Handshake Failure : SSL_accept:Error in before SSL initialization

[Kamala Ayyar <kamala.ayyar@xxxxxxxxx>]

Hello Matt,

I used the same test server application to listen on the port but used the command line version for the client to connect and it connects without issue and handshake done and server certificates displayed on screen. openssl s_client -connect servername:20333

I am not sure what is the issue with the test client application written in c++  which uses the same library and the same certificates.  I am not doing any Client Authentication. I get the same error -Error in SSLv3/TLS write client hello

The non SSL socket connects and data is exchanged.

Thanks

Kamala

On Tue, Feb 8, 2022 at 1:17 PM Kamala Ayyar <kamala.ayyar@xxxxxxxxx> wrote:

Hello Matt,

The socket descriptor is good and I verified using the socket directly.  I do exchange data between client and server successfully before passing it the function to convert to a secure socket.

It fails at the same place as the SSL_accept() with the same error  from the call back function

SSL_accept
SSL_CB_LOOP | SSL_accept:before SSL initialization

SSL_accept

SSL_accept:Error in before SSL initialization

On the client side the error is 

SSL_connect
SSL_CB_LOOP | SSL_connect:before SSL initialization

SSL_connect
SSL_connect:Error in SSLv3/TLS write client hello

I used your example and wrapped the socket with the bio however I was not able to do a bio read(). I am getting a return of -1 from the bio_read and SSL_get_error() and ERR_prints_error does not print anything

Thanks

Kamala

On Fri, Feb 4, 2022 at 1:20 PM Matt Caswell <matt@xxxxxxxxxxx> wrote:

Are you sure that the socket descriptor in "*this" is good and works?

You could test that by wrapping it in a BIO like this:

     BIO *bio = BIO_new(BIO_s_socket());

     if (bio == NULL)
        goto err;
     BIO_set_fd(bio, *this, BIO_NOCLOSE);

and then attempting to read some data from it using BIO_read(). If the
BIO_read call fails then it suggests the socket descriptor is bad.

Matt



On 04/02/2022 18:06, Kamala Ayyar wrote:
> Hello Matt,
>
> I call the WSAGetLastError() for Windows and that returns 183
> (ERROR_ALREADY_EXISTS) //Cannot create a file when that file already exists
> The SSL_get_error() gives us SSL_ERROR_SYSCALL
> *Server *code is roughly like below
> SSL_CTX *m_pCtx;
> SSL *m_pSsl;
> m_pCtx = SSL_CTX_new(TLS_server_method();
> if ((dwRet = LoadCertificates()) != rSUCCESS)
> throw dwRet;
> if ((m_pSsl = SSL_new(m_pCtx)) != NULL)
> {
>       if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the
> socket descriptor */
>      {
>      sslError = SSL_get_error(m_pSsl, iRet);
>      LOGERROR(szLine);
>      throw eSSL_ERROR;
>      }
>     SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
>     ERR_clear_error();
>     if ((sslError = SSL_accept(m_pSsl)) < 1)
>       {
>           sslError = SSL_get_error(m_pSsl, sslError);
>           dwRet = handleError(sslError, "SSL_accept failed with error ",
> iRet);
>           throw dwRet;// eSSL_ERROR;
>       }
> }
>
> Client
> SSL_CTX *m_pCtx;
> SSL *m_pSsl;
> m_pCtx = SSL_CTX_new(TLS_client_method();
> if ((dwRet = LoadCertificates(TRUE)) != rSUCCESS) //Trust certificates only
> throw dwRet;
> /* Set for server verification*/
> SSL_CTX_set_verify(m_pCtx, SSL_VERIFY_PEER, NULL); //Work in progress
> m_pSsl = SSL_new(m_pCtx);
> if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket
> descriptor */
> {
>     ssl_error = SSL_get_error(m_pSsl, iRet);
>     LOGERROR(szLine);
>     throw eSSL_ERROR;
> }
> SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
> ERR_clear_error();
> if ((iRet = SSL_connect(m_pSsl)) <= 0)   /* perform the connection */
> {
> ssl_error = SSL_get_error(m_pSsl, iRet);
> dwRet = handleError(iRet, "SSL_connect failed with error ", ssl_error);
> throw eSSL_ERROR;
> }
>
> ShowCerts();
> }
>
> As mentioned before this code works fine when  called by another
> application. So the certificates are all valid. I also tried this on
> different machines  but it did not work- I get the same error.
> Thanks
> Kamala
>
> On Fri, Feb 4, 2022 at 12:20 PM Matt Caswell <matt@xxxxxxxxxxx
> <mailto:matt@xxxxxxxxxxx>> wrote:
>
>     Does errno give you anything?
>
>     How did you create your BIOs for m_pSsl?
>
>     Matt
>
>     On 04/02/2022 16:25, Kamala Ayyar wrote:
>      > Hello Matt,
>      >
>      > The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
>      > anything for this error, just an empty string.
>      > I use the following to print error but nothing is printed
>      > if ((retVal = SSL_accept(m_pSsl)) < 1)
>      > {
>      > sslError = SSL_get_error(m_pSsl, retVal);
>      > LOGERROR(getOpenSSLError());
>      > throw dwRet;// eSSL_ERROR;
>      > }
>      > string getOpenSSLError()
>      > {
>      > BIO *bio = BIO_new(BIO_s_mem());
>      > ERR_print_errors(bio);
>      > char *buf;
>      > size_t len = BIO_get_mem_data(bio, &buf);
>      > string ret(buf, len);
>      > BIO_free(bio);
>      > return ret;
>      > }
>      >
>      > *Kamala  Ayyar*
>      > 502 Claremont Ave.
>      > Teaneck NJ 07666-2563
>      > Tel: (201)530-0861
>      >
>      >
>      > On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt@xxxxxxxxxxx
>     <mailto:matt@xxxxxxxxxxx>
>      > <mailto:matt@xxxxxxxxxxx <mailto:matt@xxxxxxxxxxx>>> wrote:
>      >
>      >
>      >
>      >     On 04/02/2022 15:17, Kamala Ayyar wrote:
>      >      >
>      >      > Hello,
>      >      >
>      >      > We are facing a strange handshake failure issue with a test
>      >     server and
>      >      > client application using OpenSSL in Windows.  We have
>     tried with
>      >     both
>      >      > 1.1.1g and 3.0.1 versions- same problem. We created a Dll to
>      >     handle the
>      >      > OpenSSL functions- where the SSL context, SSL object and
>      >     certificates
>      >      > are handled. The certificates are obtained from the
>     Windows store
>      >     and
>      >      > converted to cert and key using PKCS12_parse()
>      >      > The server accepts non secure connection from the client
>     and then
>      >     passes
>      >      > the socket to the Dll that calls the TLS_server_method() and
>      >     creates the
>      >      > SSL context, SSL object and loads the certificates for use. It
>      >     however
>      >      > fails at SSL_accept(m_pSsl). We use a call
>      >      > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback)
>     that
>      >     gave us
>      >      > the following error information
>      >      > SSL_accept:Error in before SSL initialization
>      >      > On the client side the same Dll is called with a client
>      >      > method TLS_client_method() and the error displayed
>      >     is SSL_connect:Error
>      >      > in SSLv3/TLS write client hello
>      >      > We have confirmed the certificates are good and valid.
>      >      >
>      >      > The same Dll called from a different heavily threaded
>     application
>      >     with
>      >      > over 2000+ clients works well and handshake connections
>     established
>      >      > without issues on a different port number.
>      >      >
>      >      > We have also tried to use OpenSSL methods directly
>     without using
>      >     the Dll
>      >      > but we get the same failure.  This was also used with
>     server and
>      >     client
>      >      > on the same machine as well as different machines with the
>     same
>      >      > outcome.  The non secure communication works fine between the
>      >     server and
>      >      > the client
>      >
>      >     What does SSL_get_error() report after SSL_accept() fails?
>      >
>      >     Also please dump the OpenSSL error stack when it fails, e.g.
>     using
>      >     something like ERR_print_errors_fp(stdout);
>      >
>      >     Matt
>      >
>