Re: Segfault in chan cpool_release_pool

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi Ming,


The issue is present in the SVN version of Asterisk 13, it causes a crash at least once a day. The latest backtrace is as follows:


Core was generated by `/usr/sbin/asterisk -f -vvvg -c'.

Program terminated with signal SIGSEGV, Segmentation fault.

#0  pj_atomic_dec_and_get (atomic_var=0x6) at ../src/pj/os_core_unix.c:962

962    pj_mutex_lock( atomic_var->mutex );

[Current thread is 1 (Thread 0x7f56d7841700 (LWP 4894))]

#0  0x00007f56ddf5cb29 in pj_atomic_dec_and_get (atomic_var=0x6) at ../src/pj/os_core_unix.c:962

        new_value = <optimized out>

#1  0x00007f56dfaa7b30 in pjsip_tx_data_dec_ref (tdata=0x7f5748346bd8) at ../src/pjsip/sip_transport.c:495

#2  0x00007f56dfab2f07 in tsx_shutdown (tsx=0x7f574877e218) at ../src/pjsip/sip_transaction.c:1062

#3  0x00007f56dfab3141 in tsx_set_state (tsx=0x7f574877e218, state=state@entry=PJSIP_TSX_STATE_DESTROYED, event_src_type=event_src_type@entry=PJSIP_EVENT_TIMER, event_src=0x7f574877e3d0, flag=flag@entry=0) at ../src/pjsip/sip_transaction.c:1271

        prev_state = PJSIP_TSX_STATE_TERMINATED

#4  0x00007f56dfab31ca in tsx_on_state_terminated (tsx=<optimized out>, event=<optimized out>) at ../src/pjsip/sip_transaction.c:3337

#5  0x00007f56dfab3227 in tsx_timer_callback (theap=<optimized out>, entry=0x7f574877e3d0) at ../src/pjsip/sip_transaction.c:1171

        event = {prev = 0x0, next = 0x7f56ddf5dcb7 <pj_elapsed_time+71>, type = PJSIP_EVENT_TIMER, body = {timer = {entry = 0x7f574877e3d0}, tsx_state = {src = "" = 0x7f574877e3d0, tdata = 0x7f574877e3d0, timer = 0x7f574877e3d0, status = 1215816656, data = "" tsx = 0x2a33680, prev_state = -571089626, type = 32598}, tx_msg = {tdata = 0x7f574877e3d0}, tx_error = {tdata = 0x7f574877e3d0, tsx = 0x2a33680}, rx_msg = {rdata = 0x7f574877e3d0}, user = {user1 = 0x7f574877e3d0, user2 = 0x2a33680, user3 = 0x7f56ddf5dd26 <pj_gettickcount+54>, user4 = 0x7f574877e3d0}}}

        tsx = 0x7f574877e218

#6  0x00007f56ddf6b3b7 in pj_timer_heap_poll (ht=0x2a33680, next_delay=next_delay@entry=0x7f56d7840d70) at ../src/pj/timer.c:643

        node = 0x7f574877e3d0

        grp_lock = 0x7f5700a35868

        now = {sec = 662112, msec = 455}

        count = 1

#7  0x00007f56dfaa2c3b in pjsip_endpt_handle_events2 (endpt=0x2a33398, max_timeout=max_timeout@entry=0x7f56d7840db0, p_count=p_count@entry=0x0) at ../src/pjsip/sip_endpoint.c:712

        timeout = {sec = 0, msec = 0}

        count = 0

        net_event_count = 0

        c = <optimized out>

#8  0x00007f56dfaa2d07 in pjsip_endpt_handle_events (endpt=<optimized out>, max_timeout=max_timeout@entry=0x7f56d7840db0) at ../src/pjsip/sip_endpoint.c:769

#9  0x00007f56d9e71638 in monitor_thread_exec (endpt=<optimized out>) at res_pjsip.c:3870

        delay = {sec = 0, msec = 10}

#10 0x00007f56ddf5ca56 in thread_main (param=0x2c0a4e8) at ../src/pj/os_core_unix.c:541

        rec = 0x2c0a4e8

        result = <optimized out>

#11 0x00007f576a3b161a in start_thread () at /lib64/libpthread.so.0

#12 0x00007f57696ed59d in clone () at /lib64/libc.so.6


Thread:

Thread 1 (Thread 0x7f56d7841700 (LWP 4894)):
#0  0x00007f56ddf5cb29 in pj_atomic_dec_and_get (atomic_var=0x6) at ../src/pj/os_core_unix.c:962
#1  0x00007f56dfaa7b30 in pjsip_tx_data_dec_ref (tdata=0x7f5748346bd8) at ../src/pjsip/sip_transport.c:495
#2  0x00007f56dfab2f07 in tsx_shutdown (tsx=0x7f574877e218) at ../src/pjsip/sip_transaction.c:1062
#3  0x00007f56dfab3141 in tsx_set_state (tsx=0x7f574877e218, state=state@entry=PJSIP_TSX_STATE_DESTROYED, event_src_type=event_src_type@entry=PJSIP_EVENT_TIMER, event_src=0x7f574877e3d0, flag=flag@entry=0) at ../src/pjsip/sip_transaction.c:1271
#4  0x00007f56dfab31ca in tsx_on_state_terminated (tsx=<optimized out>, event=<optimized out>) at ../src/pjsip/sip_transaction.c:3337
#5  0x00007f56dfab3227 in tsx_timer_callback (theap=<optimized out>, entry=0x7f574877e3d0) at ../src/pjsip/sip_transaction.c:1171
#6  0x00007f56ddf6b3b7 in pj_timer_heap_poll (ht=0x2a33680, next_delay=next_delay@entry=0x7f56d7840d70) at ../src/pj/timer.c:643
#7  0x00007f56dfaa2c3b in pjsip_endpt_handle_events2 (endpt=0x2a33398, max_timeout=max_timeout@entry=0x7f56d7840db0, p_count=p_count@entry=0x0) at ../src/pjsip/sip_endpoint.c:712
#8  0x00007f56dfaa2d07 in pjsip_endpt_handle_events (endpt=<optimized out>, max_timeout=max_timeout@entry=0x7f56d7840db0) at ../src/pjsip/sip_endpoint.c:769
#9  0x00007f56d9e71638 in monitor_thread_exec (endpt=<optimized out>) at res_pjsip.c:3870
#10 0x00007f56ddf5ca56 in thread_main (param=0x2c0a4e8) at ../src/pj/os_core_unix.c:541
#11 0x00007f576a3b161a in start_thread () at /lib64/libpthread.so.0
#12 0x00007f57696ed59d in clone () at /lib64/libc.so.6

George Joseph may be able to assist further in replicating and resolving the issue.

Regards,

Ross


From: pjsip <pjsip-bounces@xxxxxxxxxxxxxxx> on behalf of Ming <ming@xxxxxxxxx>
Sent: 08 July 2016 03:54
To: pjsip list
Subject: Re: Segfault in chan cpool_release_pool
 
Hi Ross,

Could you please provide us with the steps to reproduce the issue?
Preferably with our sample app, pjsua, and use the latest SVN version
as well (the do_handshake() problem was only recently rectified).

Regards,
Ming

On Wed, Jul 6, 2016 at 8:40 PM, Ross Beer <ross.beer@xxxxxxxxxxx> wrote:
> Looking at the Asterisk 13 code, the max is set to 1024 * 1024 which is
> greater than 65536:
>
>
> pj_caching_pool_init(&caching_pool, NULL, 1024 * 1024);
>
>
> The pool is created using:
>
>
> pj_pool_create(&caching_pool.factory, "SIP", 1024, 1024, NULL);
>
>
> The latest crash shows:
>
> Stack trace of thread 27450:
> #0 0x00007f682e7f3a28 raise (libc.so.6)
> #1 0x00007f682e7f562a abort (libc.so.6)
> #2 0x00007f682e836d7a __libc_message (libc.so.6)
> #3 0x00007f682e83f1ca _int_free (libc.so.6)
> #4 0x00007f682e84272c __libc_free (libc.so.6)
> #5 0x00007f67e737f6a5 cpool_release_pool (libpj.so.2)
> #6 0x00007f67ed05db48 pjsip_tx_data_dec_ref (libpjsip.so.2)
> #7 0x00007f67ed061808 on_data_sent (libpjsip.so.2)
> #8 0x00007f67ed061c23 on_connect_complete (libpjsip.so.2)
> #9 0x00007f67e737672f ioqueue_dispatch_write_event (libpj.so.2)
> #10 0x00007f67e7377c9b pj_ioqueue_poll (libpj.so.2)
> #11 0x00007f67ed058c85 pjsip_endpt_handle_events2 (libpjsip.so.2)
> #12 0x00007f67e5acf638 monitor_thread_exec (res_pjsip.so)
> #13 0x00007f67e7378a56 thread_main (libpj.so.2)
> #14 0x00007f682f58561a start_thread (libpthread.so.0)
> #15 0x00007f682e8c159d __clone (libc.so.6)
>
>
> How do I raise a ticket regarding this?
>
> Regards,
>
> Ross
>
> ________________________________
> From: pjsip <pjsip-bounces@xxxxxxxxxxxxxxx> on behalf of Ross Beer
> <ross.beer@xxxxxxxxxxx>
> Sent: 05 July 2016 20:21
>
> To: pjsip list
> Subject: Re: Segfault in chan cpool_release_pool
>
>
> Hi Anik,
>
>
> Thank you for your reply, one quick question. Where should
> pj_caching_pool_init max_capacity be defined, is there a parameter to set in
> config_site.h?
>
>
> Regards,
>
>
> Ross
>
>
> ________________________________
> From: pjsip <pjsip-bounces@xxxxxxxxxxxxxxx> on behalf of Anik Dasgupta
> <anikdg123@xxxxxxxxx>
> Sent: 05 July 2016 16:35
> To: pjsip list
> Subject: Re: Segfault in chan cpool_release_pool
>
> Hi ,
> I have worked a bit with pjnath and faced this segfault.
>
> My trace was like this ..
>
> #0  0x00000032cb632625 in raise () from /lib64/libc.so.6
> No symbol table info available.
> #1  0x00000032cb633e05 in abort () from /lib64/libc.so.6
> No symbol table info available.
> #2  0x00000032cb670537 in __libc_message () from /lib64/libc.so.6
> No symbol table info available.
> #3  0x00000032cb675f4e in malloc_printerr () from /lib64/libc.so.6
> No symbol table info available.
> #4  0x00000032cb678cf0 in _int_free () from /lib64/libc.so.6
> No symbol table info available.
> #5  0x0000000000462fa6 in cpool_release_pool ()
> No symbol table info available.
> #6  0x000000000043664a in destroy_tdata ()
> No symbol table info available.
> #7  0x000000000046672d in pj_timer_heap_poll ()
> No symbol table info available.
> #8  0x000000000040b4e4 in handle_events (max_msec=500, p_count=0x0,
> thread_index=0x6d3560) at ice.cpp:87
>         timeout = {sec = 0, msec = 0}
>         count = 0
>         c = 0
>         __PRETTY_FUNCTION__ = "pj_status_t handle_events(unsigned int,
> unsigned int*, void*)"
>         max_timeout = {sec = 0, msec = 500}
>         net_event_count = 0
>         index = 7
> #9  0x000000000040b7d7 in icedemo_worker_thread (unused=0x6d3560) at
> ice.cpp:168
>         thread_check = 7
>         thread_id = 140737198937856
>         my_set = {__bits = {16, 0 <repeats 15 times>}}
> #10 0x000000000045befa in thread_main ()
> No symbol table info available.
> #11 0x00000032cba07aa1 in start_thread () from /lib64/libpthread.so.0
> No symbol table info available.
> #12 0x00000032cb6e893d in clone () from /lib64/libc.so.6
> No symbol table info available.
> (gdb)
>
>
> 0  0x00000032cb632625 in raise () from /lib64/libc.so.6
> #1  0x00000032cb633e05 in abort () from /lib64/libc.so.6
> #2  0x00000032cb670537 in __libc_message () from /lib64/libc.so.6
> #3  0x00000032cb675f4e in malloc_printerr () from /lib64/libc.so.6
> #4  0x00000032cb678cf0 in _int_free () from /lib64/libc.so.6
> #5  0x0000000000462fa6 in cpool_release_pool ()
> #6  0x000000000043664a in destroy_tdata ()
> #7  0x000000000046672d in pj_timer_heap_poll ()
> #8  0x000000000040b4e4 in handle_events (max_msec=500, p_count=0x0,
> thread_index=0x6d3560) at ice.cpp:87
> #9  0x000000000040b7d7 in icedemo_worker_thread (unused=0x6d3560) at
> ice.cpp:168
> #10 0x000000000045befa in thread_main ()
> #11 0x00000032cba07aa1 in start_thread () from /lib64/libpthread.so.0
> #12 0x00000032cb6e893d in clone () from /lib64/libc.so.6
>
>
> I have set the pj_caching_pool_init max_capacity parameter to 65536. Since
> that I haven't got segfault in this .
> I am not sure whether this is the correct solution. But most probably it
> worked for me, since I haven't encountered it again.
>
>
> On Tue, Jul 5, 2016 at 8:32 PM, Ross Beer <ross.beer@xxxxxxxxxxx> wrote:
>>
>> Hi,
>>
>>
>> Can anyone provide any assistance here?
>>
>>
>> Kind regards,
>>
>>
>> Ross
>>
>>
>>
>> ________________________________
>> From: pjsip <pjsip-bounces@xxxxxxxxxxxxxxx> on behalf of Ross Beer
>> <ross.beer@xxxxxxxxxxx>
>> Sent: 01 July 2016 11:03
>> To: pjsip@xxxxxxxxxxxxxxx
>> Subject: Segfault in chan cpool_release_pool
>>
>>
>> Hi,
>>
>> I am currently receiving a segfault with PJSIP SVN 5359 and unable to
>> compile the latest SVN version due to the following errors:
>>
>> trunk_REV5374/third_party/lib
>>
>> /usr/src/pjproject-trunk_REV5374/pjmedia/lib/libpjmedia.so: undefined
>> reference to `pjmedia_video_format_mgr_instance'
>>
>> /usr/src/pjproject-trunk_REV5374/pjmedia/lib/libpjmedia.so: undefined
>> reference to `pjmedia_format_get_video_format_detail'
>>
>> /usr/src/pjproject-trunk_REV5374/pjmedia/lib/libpjmedia.so: undefined
>> reference to `pjmedia_get_video_format_info'
>>
>> collect2: error: ld returned 1 exit status
>>
>> make[2]: *** [../bin/pjmedia-test-x86_64-unknown-linux-gnu] Error 1
>>
>> make[2]: Leaving directory
>> `/usr/src/pjproject-trunk_REV5374/pjmedia/build'
>>
>> make[1]: *** [pjmedia-test-x86_64-unknown-linux-gnu] Error 2
>>
>> make[1]: Leaving directory
>> `/usr/src/pjproject-trunk_REV5374/pjmedia/build'
>>
>> make: *** [all] Error 1
>>
>>
>> Can you please suggest a fix for the above?
>>
>> Backtrace from segfault:
>>
>> 0  0x00007f9134077a28 in raise () from /lib64/libc.so.6
>>
>> [Current thread is 1 (Thread 0x7f90a236b700 (LWP 29614))]
>>
>> #0  0x00007f9134077a28 in raise () at /lib64/libc.so.6
>>
>> #1  0x00007f913407962a in abort () at /lib64/libc.so.6
>>
>> #2  0x00007f91340bad7a in  () at /lib64/libc.so.6
>>
>> #3  0x00007f91340c31ca in _int_free () at /lib64/libc.so.6
>>
>> #4  0x00007f91340c672c in free () at /lib64/libc.so.6
>>
>> #5  0x00007f90a8a876a5 in cpool_release_pool (pf=0x7f90a4584da0
>> <caching_pool>, pool=0x7f90d8a4ad40) at ../src/pj/pool_caching.c:259
>>
>>         pool_capacity = <optimized out>
>>
>>         i = <optimized out>
>>
>>         pool = <optimized out>
>>
>>         pf = <optimized out>
>>
>>         cp = 0x7f90a4584da0 <caching_pool>
>>
>>         pool_capacity = <optimized out>
>>
>> #6  0x00007f90aa5cbb48 in pjsip_tx_data_dec_ref (tdata=0x7f90d8a4ade8) at
>> ../src/pjsip/sip_transport.c:496
>>
>> #7  0x00007f90aa5cf808 in on_data_sent (asock=<optimized out>,
>> op_key=0x7f90d8a4ae40, bytes_sent=bytes_sent@entry=-120110) at
>> ../src/pjsip/sip_transport_tcp.c:1249
>>
>>         tcp = 0x7f90d896e538
>>
>>         tdata_op_key = 0x7f90d8a4ae40
>>
>> #8  0x00007f90aa5cfc23 in on_connect_complete (asock=<optimized out>,
>> status=<optimized out>) at ../src/pjsip/sip_transport_tcp.c:1515
>>
>>         pending_tx = 0x7f91183aecd8
>>
>>         op_key = <optimized out>
>>
>>         tcp = 0x7f90d896e538
>>
>>         addr = {addr = {sa_family = 0}, ipv4 = {sin_family = 0, sin_port =
>> 0, sin_addr = {s_addr = 0}, sin_zero = "-W\250\250\220\177\000"}, ipv6 =
>> {sin6_family = 0, sin6_port = 0, sin6_flowinfo = 0, sin6_addr = {s6_addr =
>> "-W\250\250\220\177\000\000\250cxØ \177\000", u6_addr32 = {2829604653,
>> 32656, 3631768488, 32656}}, sin6_scope_id = 2829604788}}
>>
>>         addrlen = 0
>>
>>         state_cb = <optimized out>
>>
>> #9  0x00007f90a8a7e72f in ioqueue_dispatch_write_event (ioqueue=0x30f01a0,
>> h=0x311a1d0) at ../src/pj/ioqueue_common_abs.c:286
>>
>>         status = 120110
>>
>>         has_lock = 1
>>
>>         h = 0x311a1d0
>>
>>         ioqueue = 0x30f01a0
>>
>>         rc = <optimized out>
>>
>> #10 0x00007f90a8a7fc9b in pj_ioqueue_poll (ioqueue=0x30f01a0,
>> timeout=timeout@entry=0x7f90a236ad70) at ../src/pj/ioqueue_epoll.c:800
>>
>>         i = <optimized out>
>>
>>         count = <optimized out>
>>
>>         event_cnt = 6
>>
>>         processed_cnt = 3
>>
>>         msec = 10
>>
>>         events = {{events = 29, data = "" = 0x3119e18, fd = 51486232,
>> u32 = 51486232, u64 = 51486232}}, {events = 29, data = "" = 0x3119fd8, fd
>> = 51486680, u32 = 51486680, u64 = 51486680}}, {events = 29, data = "" =
>> 0x311a550, fd = 51488080, u32 = 51488080, u64 = 51488080}}, {events = 29,
>> data = "" = 0x311a1d0, fd = 51487184, u32 = 51487184, u64 = 51487184}},
>> {events = 29, data = "" = 0x311a710, fd = 51488528, u32 = 51488528, u64 =
>> 51488528}}, {events = 29, data = "" = 0x311a390, fd = 51487632, u32 =
>> 51487632, u64 = 51487632}}, {events = 2829584897, data = "" = 0x200007f90,
>> fd = 32656, u32 = 32656, u64 = 8589967248}}, {events = 0, data = "" =
>> 0x7f90a8a80b46 <pj_atomic_dec_and_get+38>, fd = -1465382074, u32 =
>> 2829585222, u64 = 140259281603398}}, {events = 3856867848, data = "" =
>> 0xd86cfa9800007f90, fd = 32656, u32 = 32656, u64 = 15595115140419190672}},
>> {events = 32656, data = "" = 0x7, fd = 7, u32 = 7, u64 = 7}}, {events = 1,
>> data = "" = 0x0, fd = 0, u32 = 0, u64 = 0}}, {events = 0, data = "" =
>> 0x7f90a8a80509 <pj_thread_local_set+9>, fd = -1465383671, u32 = 2829583625,
>> u64 = 140259281601801}}, {events = 3631020696, data = "" =
>> 0xaa5d702300007f90, fd = 32656, u32 = 32656, u64 = 12276091454908039056}},
>> {events = 32656, data = "" = 0x7f9000000005, fd = 5, u32 = 5, u64 =
>> 140256452018181}}, {events = 472481536, data = "" = 0x1c297f0000007f91, fd
>> = 32657, u32 = 32657, u64 = 2029292745083879313}}, {events = 32657, data =""> >> {ptr = 0x7f911c297eb0, fd = 472481456, u32 = 472481456, u64 =
>> 140261219466928}}}
>>
>>         queue = {{key = 0x3119e18, event_type = WRITEABLE_EVENT}, {key =
>> 0x3119fd8, event_type = WRITEABLE_EVENT}, {key = 0x311a550, event_type =
>> WRITEABLE_EVENT}, {key = 0x311a1d0, event_type = WRITEABLE_EVENT}, {key =
>> 0x311a710, event_type = WRITEABLE_EVENT}, {key = 0x311a390, event_type =
>> WRITEABLE_EVENT}, {key = 0x7f911c297f00, event_type = 2829610171}, {key =
>> 0x3b9aca00, event_type = 2757250464}, {key = 0x7f911c297eb0, event_type =
>> 472481624}, {key = 0x7f911c297eb0, event_type = 2829612782}, {key =
>> 0x7f911c297fc8, event_type = 2829584897}, {key = 0x7f911c297fc8, event_type
>> = 2829604072}, {key = 0x7f911c297f58, event_type = 3631021136}, {key = 0x2,
>> event_type = 472481624}, {key = 0x7f90a236ad70, event_type = 2829584897},
>> {key = 0x2f53ff0, event_type = 2829644588}}
>>
>>         t1 = {u32 = {lo = 3647466018, hi = 8189}, u64 = 35175134652962}
>>
>>         t2 = {u32 = {lo = 3648834133, hi = 8189}, u64 = 35175136021077}
>>
>> #11 0x00007f90aa5c6c85 in pjsip_endpt_handle_events2 (endpt=0x2f53d08,
>> max_timeout=max_timeout@entry=0x7f90a236adb0, p_count=p_count@entry=0x0) at
>> ../src/pjsip/sip_endpoint.c:741
>>
>>         timeout = {sec = 0, msec = 10}
>>
>>         count = 2
>>
>>         net_event_count = 0
>>
>>         c = <optimized out>
>>
>> #12 0x00007f90aa5c6d07 in pjsip_endpt_handle_events (endpt=<optimized
>> out>, max_timeout=max_timeout@entry=0x7f90a236adb0) at
>> ../src/pjsip/sip_endpoint.c:769
>>
>> #13 0x00007f90a4353508 in monitor_thread_exec (endpt=<optimized out>) at
>> res_pjsip.c:3863
>>
>>         delay = {sec = 0, msec = 10}
>>
>> #14 0x00007f90a8a80a56 in thread_main (param=0x2f5e9f8) at
>> ../src/pj/os_core_unix.c:541
>>
>>         rec = 0x2f5e9f8
>>
>>         result = <optimized out>
>>
>> #15 0x00007f9134e0961a in start_thread () at /lib64/libpthread.so.0
>>
>> #16 0x00007f913414559d in clone () at /lib64/libc.so.6
>>
>>
>> Kind regards,
>>
>>
>> Ross
>>
>>
>>
>> _______________________________________________
>> Visit our blog: http://blog.pjsip.org
>>
>> pjsip mailing list
>> pjsip@xxxxxxxxxxxxxxx
>> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>>
>
>
> _______________________________________________
> Visit our blog: http://blog.pjsip.org
>
> pjsip mailing list
> pjsip@xxxxxxxxxxxxxxx
> http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
>

_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@xxxxxxxxxxxxxxx
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org
_______________________________________________
Visit our blog: http://blog.pjsip.org

pjsip mailing list
pjsip@xxxxxxxxxxxxxxx
http://lists.pjsip.org/mailman/listinfo/pjsip_lists.pjsip.org

[Index of Archives]     [Asterisk Users]     [Asterisk App Development]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Security]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [Linux API]
  Powered by Linux