Re: How to fix "OpenSSL failed - error:0A000086:SSL routines::certificate verify failed"

[Pierre-Luc Boily <pierreluc.boily@xxxxxxxxx>]

I am asking a question regarding OpenSSL.  I thought the mailing list was the place.  I read this on the github page of OpenSSL

    If you have questions about how to use OpenSSL for specific tasks
    or how to solve certain problems you have when using it, you might
    want to ask them on the openssl-users@xxxxxxxxxxx mailing list.
    There you can get help from a great community of OpenSSL users,
    not only (but including) the OpenSSL developers. For more information
    about our mailing lists, see
    https://www.openssl.org/community/mailinglists.html.

Le ven. 16 déc. 2022, à 17 h 14, psv sridhar <psv_sridhar@xxxxxxxxx> a écrit :

you are sending flooded emails wrongly. stop it.

 

Thanks and Regards
Sridhar PSV

Phone 571 244-5862

On Friday, December 16, 2022 at 04:08:38 PM CST, Pierre-Luc Boily <pierreluc.boily@xxxxxxxxx> wrote:

Hello,

Details

OS : WIndows 10

Arch : x64

Compiler : VisualStudio 2017

I have a c++ wss IXWebSocket client that tries to connect to a nodejs https/websocket server but the client refuses to connect and returns the error : OpenSSL failed - error:0A000086:SSL routines::certificate verify failed

What I tried

1. I have a React front end using wss to communicate to my https nodejs server. It works -> This confirms that my key and certificate are valid.
2. I also tried the same c++ client above, not secured (no wss) connecting to my same nodejs server, but http/websocket (non secure). It works.

So, I had to dig into the OpenSSL code and I found where the error is triggered, see code below.  In my case s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 0 and I don't know if those values are OK or not.

While I was digging into the code, I also realized that SSL_OP_NO_TLSv1_3 is automagically defined for my code.  I feel that it is incorrect.

From statem_clnt.c line 1888:

    if (s->verify_mode != SSL_VERIFY_NONE && i <= 0) {
        SSLfatal(s, ssl_x509err2alert(s->verify_result),
                 SSL_R_CERTIFICATE_VERIFY_FAILED);
        return WORK_ERROR;
    }

Stacktrace:

> libssl-3-x64.dll!tls_post_process_server_certificate(ssl_st
  libssl-3-x64.dll!ossl_statem_client_post_process_message(ss
  libssl-3-x64.dll!read_state_machine(ssl_st * s) Line 675
  libssl-3-x64.dll!state_machine(ssl_st * s, int server) Line
  libssl-3-x64.dll!ossl_statem_connect(ssl_st * s) Line 266
  libssl-3-x64.dll!SSL_do_handshake(ssl_st * s) Line 3937 C  
  libssl-3-x64.dll!SSL_connect(ssl_st * s) Line 1760 C      
  testWSClient.exe!ix::SocketOpenSSL::openSSLClientHandshake(
  testWSClient.exe!ix::SocketOpenSSL::connect(const std::basi
  testWSClient.exe!ix::WebSocketHandshake::clientHandshake(co
  testWSClient.exe!ix::WebSocketTransport::connectToUrl(const
  testWSClient.exe!ix::WebSocket::connect(int timeoutSecs) Li
  testWSClient.exe!ix::WebSocket::checkConnection(bool firstC
  testWSClient.exe!ix::WebSocket::run() Line 367 C++        

IXWebClient, how key/cert are set :

    ix::SocketTLSOptions tlsOptions;
    tlsOptions.certFile = "WebRTC.test.crt";
    tlsOptions.keyFile = "WebRTC.test.key";
    tlsOptions.caFile = "WebRTC-CA.pem";
    webSocket.setTLSOptions(tlsOptions);
    std::string url("wss://localhost:8080");
    webSocket.setUrl(url);

No matter if the path of the key/certificate exists or not, I have the same error message from OpenSSL, which is weird...

So : 

1. Any idea why I have certificate verify failed?

2. Is it normal that s->verify_mode is equal to SSL_VERIFY_PEER and i equal to 0

3. Is it normal that SSL_OP_NO_TLSv1_3 is enabled in the code?

Thanks a lot for any help.