Does errno give you anything?
How did you create your BIOs for m_pSsl?
Matt
On 04/02/2022 16:25, Kamala Ayyar wrote:
Hello Matt,
The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
anything for this error, just an empty string.
I use the following to print error but nothing is printed
if ((retVal = SSL_accept(m_pSsl)) < 1)
{
sslError = SSL_get_error(m_pSsl, retVal);
LOGERROR(getOpenSSLError());
throw dwRet;// eSSL_ERROR;
}
string getOpenSSLError()
{
BIO *bio = BIO_new(BIO_s_mem());
ERR_print_errors(bio);
char *buf;
size_t len = BIO_get_mem_data(bio, &buf);
string ret(buf, len);
BIO_free(bio);
return ret;
}
*Kamala Ayyar*
502 Claremont Ave.
Teaneck NJ 07666-2563
Tel: (201)530-0861
On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt@xxxxxxxxxxx
<mailto:matt@xxxxxxxxxxx>> wrote:
On 04/02/2022 15:17, Kamala Ayyar wrote:
>
> Hello,
>
> We are facing a strange handshake failure issue with a test
server and
> client application using OpenSSL in Windows. We have tried with
both
> 1.1.1g and 3.0.1 versions- same problem. We created a Dll to
handle the
> OpenSSL functions- where the SSL context, SSL object and
certificates
> are handled. The certificates are obtained from the Windows store
and
> converted to cert and key using PKCS12_parse()
> The server accepts non secure connection from the client and then
passes
> the socket to the Dll that calls the TLS_server_method() and
creates the
> SSL context, SSL object and loads the certificates for use. It
however
> fails at SSL_accept(m_pSsl). We use a call
> back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback) that
gave us
> the following error information
> SSL_accept:Error in before SSL initialization
> On the client side the same Dll is called with a client
> method TLS_client_method() and the error displayed
is SSL_connect:Error
> in SSLv3/TLS write client hello
> We have confirmed the certificates are good and valid.
>
> The same Dll called from a different heavily threaded application
with
> over 2000+ clients works well and handshake connections established
> without issues on a different port number.
>
> We have also tried to use OpenSSL methods directly without using
the Dll
> but we get the same failure. This was also used with server and
client
> on the same machine as well as different machines with the same
> outcome. The non secure communication works fine between the
server and
> the client
What does SSL_get_error() report after SSL_accept() fails?
Also please dump the OpenSSL error stack when it fails, e.g. using
something like ERR_print_errors_fp(stdout);
Matt
