Re: How to fix "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"

[psv sridhar via openssl-users <openssl-users@xxxxxxxxxxx>]

you are sending flooded emails wrongly. stop it.

 

Thanks and Regards
Sridhar PSV

Phone 571 244-5862

On Friday, December 16, 2022 at 04:08:38 PM CST, Pierre-Luc Boily <pierreluc.boily@xxxxxxxxx> wrote:

Hello,

Details

OS : WIndows 10

Arch : x64

Compiler : VisualStudio 2017

I have a c++ wss IXWebSocket client that tries to connect to a nodejs https/websocket server but the client refuses to connect and returns the error : OpenSSL failed - error:0A000086:SSL routines::certificate verify failed

What I tried

1. I have a React front end using wss to communicate to my https nodejs server. It works -> This confirms that my key and certificate are valid.
2. I also tried the same c++ client above, not secured (no wss) connecting to my same nodejs server, but http/websocket (non secure). It works.

So, I had to dig into the OpenSSL code and I found where the error is triggered, see code below.  In my case s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 0 and I don't know if those values are OK or not.

While I was digging into the code, I also realized that SSL_OP_NO_TLSv1_3 is automagically defined for my code.  I feel that it is incorrect.

From statem_clnt.c line 1888:

    if (s->verify_mode != SSL_VERIFY_NONE && i <= 0) {
        SSLfatal(s, ssl_x509err2alert(s->verify_result),
                 SSL_R_CERTIFICATE_VERIFY_FAILED);
        return WORK_ERROR;
    }

Stacktrace:

> libssl-3-x64.dll!tls_post_process_server_certificate(ssl_st
  libssl-3-x64.dll!ossl_statem_client_post_process_message(ss
  libssl-3-x64.dll!read_state_machine(ssl_st * s) Line 675
  libssl-3-x64.dll!state_machine(ssl_st * s, int server) Line
  libssl-3-x64.dll!ossl_statem_connect(ssl_st * s) Line 266
  libssl-3-x64.dll!SSL_do_handshake(ssl_st * s) Line 3937 C  
  libssl-3-x64.dll!SSL_connect(ssl_st * s) Line 1760 C      
  testWSClient.exe!ix::SocketOpenSSL::openSSLClientHandshake(
  testWSClient.exe!ix::SocketOpenSSL::connect(const std::basi
  testWSClient.exe!ix::WebSocketHandshake::clientHandshake(co
  testWSClient.exe!ix::WebSocketTransport::connectToUrl(const
  testWSClient.exe!ix::WebSocket::connect(int timeoutSecs) Li
  testWSClient.exe!ix::WebSocket::checkConnection(bool firstC
  testWSClient.exe!ix::WebSocket::run() Line 367 C++        

IXWebClient, how key/cert are set :

    ix::SocketTLSOptions tlsOptions;
    tlsOptions.certFile = "WebRTC.test.crt";
    tlsOptions.keyFile = "WebRTC.test.key";
    tlsOptions.caFile = "WebRTC-CA.pem";
    webSocket.setTLSOptions(tlsOptions);
    std::string url("wss://localhost:8080");
    webSocket.setUrl(url);

No matter if the path of the key/certificate exists or not, I have the same error message from OpenSSL, which is weird...

So : 

1. Any idea why I have certificate verify failed?

2. Is it normal that s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 0

3. Is it normal that SSL_OP_NO_TLSv1_3 is enabled in the code?

Thanks a lot for any help.