OpenSSL 1.1.0 assertion failure: ssl_free_wbio_buffer()

[Marcus.Schafheutle@xxxxxx]

Hello,
 
we encountered a strange behaviour of OpenSSL 1.1.0 when our test with "sslscan" provokes an unfinished 
handshake. The problem exists since OpenSSL 1.1.0 - there were no problems with OpenSSL 1.0.2x.
 
Our asynchronous communication approach uses memory BIOs in order to read/write data from other 
communication layers into the SSL object. After the read/write operations are done, the BIOs are freed.
 
Our example code is basically as follows:
---
err = SSL_accept (ssl); 
 
BIO *rBIO = BIO_new(BIO_s_mem());
BIO *wBIO = BIO_new(BIO_s_mem());
    
SSL_set_bio(ssl, rBIO, wBIO); 
 
// ... operate with read / write BIOs and SSL_read/SSL_write
 
SSL_set_bio(ssl, NULL, NULL); // free BIOs when finished
 
SSL_free(ssl);
---
 
When calling SSL_free() after a complete handshake, everything is fine, but when calling SSL_free() after an 
unfinished handshake, the assertion "assert(s->wbio != NULL);" in ssl_free_wbio_buffer() fails:
---
void ssl_free_wbio_buffer(SSL *s)
{
    /* callers ensure s is never null */
    if (s->bbio == NULL)
        return;
    s->wbio = BIO_pop(s->wbio);
    assert(s->wbio != NULL);            /* <- this assertion fails! */
    BIO_free(s->bbio);
    s->bbio = NULL;
}
---
 
With a complete handshake the new attribute “bbio” is freed by calling the method tls_finish_handshake(), 
which itself calls ssl_free_wbio_buffer(). When the handshake is not finished successfully, the “bbio” is not freed, and 
therefore when calling SSL_free() the assert in ssl_free_wbio_buffer() fails.
  
Is this a faulty behavior of OpenSSL 1.1.0 or is the fault on my side? 
 
Thank you for your help!
 
Best regards,
Marcus

-- 
openssl-users mailing list
To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users