Random and rare Seg faults at openssl library level

[Gimhani Uthpala <gimhanieuthpala@xxxxxxxxx>]

Dear team,

I'm running an application which uses openssl for secure communication between processes. I am getting seg-faults at openssl level. This only occurred very randomly and the following are stacks that seg faults  at openssl level in the given 2 cases. We are using openssl 1.0.2k. 

Went through the security vulnerabilities list for this version but couldn't find a clue. Running valgrind too didn't give an exact clue related to the issue. Can you please guide me how can I find the exact root cause for the seg fault?  

I am calling SSL_do_handshake(ssl_ctx) from my code level and both the below seg faults are occuring from it's inside. 

#0  0x00007fd64cdabdd3 in ASN1_item_verify () from /lib64/libcrypto.so.10
#1  0x00007fd64cdcac58 in internal_verify () from /lib64/libcrypto.so.10
#2  0x00007fd64cdccaef in X509_verify_cert () from /lib64/libcrypto.so.10
#3  0x00007fd64d111c68 in ssl_verify_cert_chain () from /lib64/libssl.so.10
#4  0x00007fd64d0e8cc6 in ssl3_get_client_certificate () from /lib64/libssl.so.10
#5  0x00007fd64d0ea3f8 in ssl3_accept () from /lib64/libssl.so.10

 #0 0x00007ffb65529854 in RSA_verify () from /usr/lib64/libcrypto.so.10
 #1 0x00007ffb6552f1fc in pkey_rsa_verify () from /usr/lib64/libcrypto.so.10
 #2 0x00007ffb65561877 in EVP_DigestVerifyFinal () from /usr/lib64/libcrypto.so.10
 #3 0x00007ffb6556af25 in ASN1_item_verify () from /usr/lib64/libcrypto.so.10
 #4 0x00007ffb65589c58 in internal_verify () from /usr/lib64/libcrypto.so.10
 #5 0x00007ffb6558baef in X509_verify_cert () from /usr/lib64/libcrypto.so.10
 #6 0x00007ffb658d1417 in ssl_add_cert_chain () from /usr/lib64/libssl.so.10
 #7 0x00007ffb658b6095 in ssl3_output_cert_chain () from /usr/lib64/libssl.so.10
 #8 0x00007ffb658ae894 in ssl3_send_client_certificate () from /usr/lib64/libssl.so.10
 #9 0x00007ffb658aeecf in ssl3_connect () from /usr/lib64/libssl.so.10
 #10 0x00007ffb658b8e7e in ssl23_connect () from /usr/lib64/libssl.so.10

  I am setting context to use SSLv23_method() s. However, I can see ssl3_ methods being called. Is there any issue with that?  

Given below is SSL* object passed to SSL_do_handshake method.

(gdb) p *p_SSLCtx
$2 = {version = 771, type = 4096, method = 0x7ffb65af3320, rbio = 0x7ffb5819e140, wbio = 0x7ffb58193570, bbio = 0x7ffb58193570, rwstate = 1, in_handshake = 2, handshake_func = 0x7ffb658aea30 <ssl3_connect>, server = 0, new_session = 0, quiet_shutdown = 0, shutdown = 0, state = 4467,
  rstate = 240, init_buf = 0x7ffb581934b0, init_msg = 0x7ffb581e10d4, init_num = 0, init_off = 0, packet = 0x7ffb581e6633 "\026\003\003", packet_length = 0, s2 = 0x0, s3 = 0x7ffb58195ee0, d1 = 0x0, read_ahead = 0, msg_callback = 0x0, msg_callback_arg = 0x0, hit = 0, param = 0x7ffb581933f0,
  cipher_list = 0x0, cipher_list_by_id = 0x0, mac_flags = 0, enc_read_ctx = 0x0, read_hash = 0x0, expand = 0x0, enc_write_ctx = 0x0, write_hash = 0x0, compress = 0x0, cert = 0x7ffb58195ba0, sid_ctx_length = 0, sid_ctx = '\000' <repeats 31 times>, session = 0x7ffb58198100,
  generate_session_id = 0x0,......

 

--

Gimhani Uthpala Kankanamge