Dear PJSIP experts,
I'm running pjsip-2.7.1 on armhf, and got a crash today that looks like a race condition in PJSIP. This is a one-time occurrence from our stress tests and has not been reproduced. But I analyzed the coredump and logs to get below information, hoping that we might find a possible issue.
My application was running a coverage build, and made a normal outgoing call - timings included:
17:30:08.248242 |Call CALLING(1) to sip:1021@10.199.70.8
17:30:08.634752 |Call EARLY(3) to sip:1021@10.199.70.8
17:30:09.442568 |Call CONNECTING(4) to sip:1021@10.199.70.8
At this point, it ran into a SEGV because the transport (tp) was still NULL here:
- #0 0x76ceaa40 in pjmedia_transport_send_rtcp (size=64, pkt=0x663dee04, tp=0x0) at ../include/pjmedia/transport.h:792
792 return (*tp->op->send_rtcp)(tp, pkt, size);
- (gdb) bt
- #0 0x76ceaa40 in pjmedia_transport_send_rtcp (size=64, pkt=0x663dee04, tp=0x0) at ../include/pjmedia/transport.h:792
- #1 send_rtcp (stream=stream@entry=0x71f0fc4c, with_sdes=<optimized out>, with_bye=0, with_xr=0) at ../src/pjmedia/stream.c:1041
- #2 0x76ceacd0 in on_rx_rtp (data="" pkt=<optimized out>, bytes_read=<optimized out>) at ../src/pjmedia/stream.c:1882
- #3 0x76cf7c88 in srtp_rtp_cb (user_data=0x2807234, pkt=0x27ef06c, size=172) at ../src/pjmedia/transport_srtp.c:1236
- #4 0x76cf89d0 in on_rx_rtp (key=<optimized out>, op_key=<optimized out>, bytes_read=172) at ../src/pjmedia/transport_udp.c:540
- #5 0x76bca758 in ioqueue_dispatch_read_event (ioqueue=<optimized out>, h=0x28460d4) at ../src/pj/ioqueue_common_abs.c:605
- #6 0x76bcc1c8 in pj_ioqueue_poll (ioqueue=0x27e99b4, timeout=timeout@entry=0x7321ad80) at ../src/pj/ioqueue_epoll.c:812
- #7 0x76cd2c64 in worker_proc (arg=0x2849044) at ../src/pjmedia/endpoint.c:339
- #8 0x76bcdbec in thread_main (param=0x27df418) at ../src/pj/os_core_unix.c:541
- #9 0x769d707c in start_thread (arg=0x7321b3c0) at pthread_create.c:339
There was another Thread active in PJSIP, and I noticed that its frame #5 would have assigned the transport just 5 statements later in stream.c:2363 if it had not already crashed:
- #0 transport_attach2 (tp=0x2807234, param=<optimized out>) at ../src/pjmedia/transport_srtp.c:1069
- #1 0x76cee030 in pjmedia_transport_attach2 (att_param=0x6fefde20, tp=0x2807234) at ../include/pjmedia/transport.h:688
- #2 pjmedia_stream_create (endpt=<optimized out>, pool=<optimized out>, pool@entry=0x0, info=info@entry=0x6fefe068, tp=0x2807234, user_data=0x0,
- p_stream=0x76e4b4b0 <pjsua_var+22496>) at ../src/pjmedia/stream.c:2363
- #3 0x76e24810 in pjsua_aud_channel_update (call_med=call_med@entry=0x76e4b498 <pjsua_var+22472>, tmp_pool=<optimized out>, si=si@entry=0x6fefe068,
- local_sdp=local_sdp@entry=0x28138e4, remote_sdp=remote_sdp@entry=0x71f16db4) at ../src/pjsua-lib/pjsua_aud.c:624
- #4 0x76e1cc7c in pjsua_media_channel_update (call_id=<optimized out>, local_sdp=0x28138e4, remote_sdp=0x71f16db4)
- at ../src/pjsua-lib/pjsua_media.c:3092
- #5 0x76e0eef4 in pjsua_call_on_media_update (inv=0x28505cc, status=<optimized out>) at ../src/pjsua-lib/pjsua_call.c:4008
- #6 0x76dce0dc in inv_negotiate_sdp (inv=0x28505cc) at ../src/pjsip-ua/sip_inv.c:1948
- #7 0x76dcf1d0 in inv_check_sdp_in_incoming_msg (inv=inv@entry=0x28505cc, tsx=tsx@entry=0x71f1a02c, rdata=0x71f24f0c)
- at ../src/pjsip-ua/sip_inv.c:2163
- #8 0x76dd59d8 in inv_on_state_early (inv=0x28505cc, e=0x6fefe73c) at ../src/pjsip-ua/sip_inv.c:4448
- #9 0x76dce67c in mod_inv_on_tsx_state (tsx=0x71f1a02c, e=0x6fefe73c) at ../src/pjsip-ua/sip_inv.c:723
- #10 0x76d83af8 in pjsip_dlg_on_tsx_state (dlg=0x27f92bc, tsx=0x71f1a02c, e=0x6fefe73c) at ../src/pjsip/sip_dialog.c:2066
- #11 0x76d7bd38 in tsx_set_state (tsx=tsx@entry=0x71f1a02c, state=state@entry=PJSIP_TSX_STATE_TERMINATED,
- event_src_type=event_src_type@entry=PJSIP_EVENT_RX_MSG, event_src=0x71f24f0c, flag=flag@entry=0) at ../src/pjsip/sip_transaction.c:1267
- #12 0x76d7dad4 in tsx_on_state_proceeding_uac (tsx=0x71f1a02c, event=0x6fefe7cc) at ../src/pjsip/sip_transaction.c:2974
- #13 0x76d80350 in pjsip_tsx_recv_msg (tsx=tsx@entry=0x71f1a02c, rdata=rdata@entry=0x71f24f0c) at ../src/pjsip/sip_transaction.c:1827
- #14 0x76d80400 in mod_tsx_layer_on_rx_response (rdata=0x71f24f0c) at ../src/pjsip/sip_transaction.c:893
- #15 0x76d66a2c in pjsip_endpt_process_rx_data (endpt=0x27b43ac, rdata=0x71f24f0c, p=<optimized out>, p_handled=0x6fefe8a4)
- at ../src/pjsip/sip_endpoint.c:901
- #16 0x76d66bd4 in endpt_on_rx_msg (endpt=0x27b43ac, status=<optimized out>, rdata=0x71f24f0c) at ../src/pjsip/sip_endpoint.c:1043
- #17 0x76d6e000 in pjsip_tpmgr_receive_packet (mgr=0x282e4a0, rdata=rdata@entry=0x71f24f0c) at ../src/pjsip/sip_transport.c:1965
- #18 0x76d70dc0 in udp_on_read_complete (key=0x282c328, op_key=<optimized out>, bytes_read=1001) at ../src/pjsip/sip_transport_udp.c:170
- #19 0x76bca7b4 in ioqueue_dispatch_read_event (ioqueue=<optimized out>, h=0x282c328) at ../src/pj/ioqueue_common_abs.c:605
- #20 0x76bcc1c8 in pj_ioqueue_poll (ioqueue=0x282c078, timeout=timeout@entry=0x6fefed38) at ../src/pj/ioqueue_epoll.c:812
- #21 0x76d666fc in pjsip_endpt_handle_events2 (endpt=0x27b43ac, max_timeout=max_timeout@entry=0x6fefed70, p_count=p_count@entry=0x6fefed6c)
- at ../src/pjsip/sip_endpoint.c:744
- #22 0x76e11704 in pjsua_handle_events (msec_timeout=<optimized out>) at ../src/pjsua-lib/pjsua_core.c:1988
- #23 0x76e11744 in worker_thread (arg=<optimized out>) at ../src/pjsua-lib/pjsua_core.c:704
- #24 0x76bcdbec in thread_main (param=0x27d3b44) at ../src/pj/os_core_unix.c:541
- #25 0x769d707c in start_thread (arg=0x6feff3c0) at pthread_create.c:339
I have checked my application, and I believe that none of my callbacks are involved here. Also noticed from the Coredump that PjCall::onStreamCreated() had not been called yet.
These facts are making me believe that I might have found a race condition or insufficient lock protection in PJSIP. Is this ringing a bell for anyone, has such a problem been seen before? - I have searched existing issues on Github and not found anything related, but maybe some of the experts could shed some light on this.
PS I cannot currently upgrade from pjsip-2.7.1 :( so would appreciate any ideas that might point me in the right direction!
Thanks a lot,
Martin
_______________________________________________ Visit our blog: http://blog.pjsip.org pjsip mailing list pjsip@xxxxxxxxxxxxxxx http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
