Are you sure that the socket descriptor in "*this" is good and works?
You could test that by wrapping it in a BIO like this:
BIO *bio = BIO_new(BIO_s_socket());
if (bio == NULL)
goto err;
BIO_set_fd(bio, *this, BIO_NOCLOSE);
and then attempting to read some data from it using BIO_read(). If the
BIO_read call fails then it suggests the socket descriptor is bad.
Matt
On 04/02/2022 18:06, Kamala Ayyar wrote:
Hello Matt,
I call the WSAGetLastError() for Windows and that returns 183
(ERROR_ALREADY_EXISTS) //Cannot create a file when that file already exists
The SSL_get_error() gives us SSL_ERROR_SYSCALL
*Server *code is roughly like below
SSL_CTX *m_pCtx;
SSL *m_pSsl;
m_pCtx = SSL_CTX_new(TLS_server_method();
if ((dwRet = LoadCertificates()) != rSUCCESS)
throw dwRet;
if ((m_pSsl = SSL_new(m_pCtx)) != NULL)
{
if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the
socket descriptor */
{
sslError = SSL_get_error(m_pSsl, iRet);
LOGERROR(szLine);
throw eSSL_ERROR;
}
SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
ERR_clear_error();
if ((sslError = SSL_accept(m_pSsl)) < 1)
{
sslError = SSL_get_error(m_pSsl, sslError);
dwRet = handleError(sslError, "SSL_accept failed with error ",
iRet);
throw dwRet;// eSSL_ERROR;
}
}
Client
SSL_CTX *m_pCtx;
SSL *m_pSsl;
m_pCtx = SSL_CTX_new(TLS_client_method();
if ((dwRet = LoadCertificates(TRUE)) != rSUCCESS) //Trust certificates only
throw dwRet;
/* Set for server verification*/
SSL_CTX_set_verify(m_pCtx, SSL_VERIFY_PEER, NULL); //Work in progress
m_pSsl = SSL_new(m_pCtx);
if ((iRet = SSL_set_fd(m_pSsl, (*this)())) == 0) /* attach the socket
descriptor */
{
ssl_error = SSL_get_error(m_pSsl, iRet);
LOGERROR(szLine);
throw eSSL_ERROR;
}
SSL_set_info_callback(m_pSsl, apps_ssl_info_callback);
ERR_clear_error();
if ((iRet = SSL_connect(m_pSsl)) <= 0) /* perform the connection */
{
ssl_error = SSL_get_error(m_pSsl, iRet);
dwRet = handleError(iRet, "SSL_connect failed with error ", ssl_error);
throw eSSL_ERROR;
}
ShowCerts();
}
As mentioned before this code works fine when called by another
application. So the certificates are all valid. I also tried this on
different machines but it did not work- I get the same error.
Thanks
Kamala
On Fri, Feb 4, 2022 at 12:20 PM Matt Caswell <matt@xxxxxxxxxxx
<mailto:matt@xxxxxxxxxxx>> wrote:
Does errno give you anything?
How did you create your BIOs for m_pSsl?
Matt
On 04/02/2022 16:25, Kamala Ayyar wrote:
> Hello Matt,
>
> The SSL_get_error() returns 5(SSL_ERROR_SYSCALL) It does not print
> anything for this error, just an empty string.
> I use the following to print error but nothing is printed
> if ((retVal = SSL_accept(m_pSsl)) < 1)
> {
> sslError = SSL_get_error(m_pSsl, retVal);
> LOGERROR(getOpenSSLError());
> throw dwRet;// eSSL_ERROR;
> }
> string getOpenSSLError()
> {
> BIO *bio = BIO_new(BIO_s_mem());
> ERR_print_errors(bio);
> char *buf;
> size_t len = BIO_get_mem_data(bio, &buf);
> string ret(buf, len);
> BIO_free(bio);
> return ret;
> }
>
> *Kamala Ayyar*
> 502 Claremont Ave.
> Teaneck NJ 07666-2563
> Tel: (201)530-0861
>
>
> On Fri, Feb 4, 2022 at 10:54 AM Matt Caswell <matt@xxxxxxxxxxx
<mailto:matt@xxxxxxxxxxx>
> <mailto:matt@xxxxxxxxxxx <mailto:matt@xxxxxxxxxxx>>> wrote:
>
>
>
> On 04/02/2022 15:17, Kamala Ayyar wrote:
> >
> > Hello,
> >
> > We are facing a strange handshake failure issue with a test
> server and
> > client application using OpenSSL in Windows. We have
tried with
> both
> > 1.1.1g and 3.0.1 versions- same problem. We created a Dll to
> handle the
> > OpenSSL functions- where the SSL context, SSL object and
> certificates
> > are handled. The certificates are obtained from the
Windows store
> and
> > converted to cert and key using PKCS12_parse()
> > The server accepts non secure connection from the client
and then
> passes
> > the socket to the Dll that calls the TLS_server_method() and
> creates the
> > SSL context, SSL object and loads the certificates for use. It
> however
> > fails at SSL_accept(m_pSsl). We use a call
> > back SSL_set_info_callback(m_pSsl, apps_ssl_info_callback)
that
> gave us
> > the following error information
> > SSL_accept:Error in before SSL initialization
> > On the client side the same Dll is called with a client
> > method TLS_client_method() and the error displayed
> is SSL_connect:Error
> > in SSLv3/TLS write client hello
> > We have confirmed the certificates are good and valid.
> >
> > The same Dll called from a different heavily threaded
application
> with
> > over 2000+ clients works well and handshake connections
established
> > without issues on a different port number.
> >
> > We have also tried to use OpenSSL methods directly
without using
> the Dll
> > but we get the same failure. This was also used with
server and
> client
> > on the same machine as well as different machines with the
same
> > outcome. The non secure communication works fine between the
> server and
> > the client
>
> What does SSL_get_error() report after SSL_accept() fails?
>
> Also please dump the OpenSSL error stack when it fails, e.g.
using
> something like ERR_print_errors_fp(stdout);
>
> Matt
>
