Hi, I updated the ocserv to commit 3eb2b903ed8c9cd4e10304e319465e1e50424f97(worker: Prevent any frozen worker processes by killing them on inactivity), and found sometimes the ocserv worker still become inactive. #A For those "active" connections, there is no more effective requests(cannot ping gateway), even though by tcpdump it's seen there was still some repeated communication between the server and the client, then after couples of minutes the problem was gone and a reconnection was launched automatically. #B For those already disconnections, the occtl cannot disconnect them as before said, once again several minutes later this worker was destoryed automatically. Please have a check with both. #A "Active" wokrer gdb: gdb /usr/bin/ocserv 16553 GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... /usr/bin/ocserv: No such file or directory. Attaching to process 16553 Reading symbols from /usr/local/sbin/ocserv...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libopts.so.25...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libopts.so.25 Reading symbols from /usr/local/lib/libgnutls.so.28...done. Loaded symbols for /usr/local/lib/libgnutls.so.28 Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libpam.so.0 Reading symbols from /usr/lib/x86_64-linux-gnu/libseccomp.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libseccomp.so.2 Reading symbols from /lib/x86_64-linux-gnu/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libwrap.so.0 Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1 Reading symbols from /lib/libfreeradius-client.so.2...done. Loaded symbols for /lib/libfreeradius-client.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libnettle.so.4 Reading symbols from /usr/lib/x86_64-linux-gnu/libev.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libev.so.4 Reading symbols from /usr/lib/libpcl.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libpcl.so.1 Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6 Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libhogweed.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libgmp.so.10 Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libaudit.so.1 Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6 Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_compat-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_nis-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 0x00007fac5291de1b in __pselect (nfds=15, readfds=readfds at entry=0x7ffff8899e80, writefds=writefds at entry=0x0, exceptfds=exceptfds at entry=0x0, timeout=<optimized out>, timeout at entry=0x7ffff8899e40, sigmask=sigmask at entry=0x7ffff8899f00) at ../sysdeps/unix/sysv/linux/pselect.c:71 71 ../sysdeps/unix/sysv/linux/pselect.c: No such file or directory. (gdb) bt full #0 0x00007fac5291de1b in __pselect (nfds=15, readfds=readfds at entry=0x7ffff8899e80, writefds=writefds at entry=0x0, exceptfds=exceptfds at entry=0x0, timeout=<optimized out>, timeout at entry=0x7ffff8899e40, sigmask=sigmask at entry=0x7ffff8899f00) at ../sysdeps/unix/sysv/linux/pselect.c:71 resultvar = 18446744073709551102 tval = {tv_sec = 9, tv_nsec = 126225802} data = {ss = 140737363156736, ss_len = 8} result = <optimized out> #1 0x000000000040b6b4 in connect_handler (ws=ws at entry=0xff9c30) at worker-vpn.c:1922 req = 0xff9c60 rfds = {fds_bits = {16387, 0 <repeats 15 times>}} e = <optimized out> max = 14 ret = <optimized out> t = 2048 p = <optimized out> rnd = 1335569060 tv = {tv_sec = 10, tv_nsec = 0} tls_pending = 0 dtls_pending = 0 i = <optimized out> tnow = {tv_sec = 1453020235, tv_nsec = 813061000} ip6 = <optimized out> sl = 4 emptyset = {__val = {0 <repeats 16 times>}} blockset = {__val = {16384, 0 <repeats 15 times>}} #2 0x000000000040d69c in vpn_server (ws=ws at entry=0xff9c30) at worker-vpn.c:571 ret = <optimized out> nparsed = <optimized out> nrecvd = <optimized out> session = 0x1017010 parser = {type = 0 '\000', flags = 0 '\000', state = 17 '\021', header_state = 0 '\000', index = 0 '\000', nread = 0, content_length = 18446744073709551615, http_major = 1, http_minor = 1, status_code = 0, method = 5 '\005', http_errno = 0 '\000', upgrade = 1 '\001', data = 0xff9c30} settings = {on_message_begin = 0x0, on_url = 0x41db60 <http_url_cb>, on_status_complete = 0x0, on_header_field = 0x41dbb0 <http_header_field_cb>, on_header_value = 0x41dcc0 <http_header_value_cb>, on_headers_complete = 0x41ddc0 <http_header_complete_cb>, on_body = 0x41de50 <http_body_cb>, on_message_complete = 0x41de30 <http_message_complete_cb>} fn = <optimized out> requests_left = 15 #3 0x0000000000408b87 in listen_watcher_cb (loop=0x7fac53002900, w=<optimized out>, revents=<optimized out>) at main.c:1061 stype = 0 s = 0xfe59e0 ltmp = <optimized out> ctmp = 0x0 ws = 0xff9c30 fd = 0 ret = <optimized out> cmd_fd = {13, 14} pid = <optimized out> #4 0x00007fac52df8e45 in ev_invoke_pending () from /usr/lib/x86_64-linux-gnu/libev.so.4 No symbol table info available. #5 0x00007fac52dfbfe7 in ev_run () from /usr/lib/x86_64-linux-gnu/libev.so.4 ---Type <return> to continue, or q <return> to quit--- No symbol table info available. #6 0x0000000000406b42 in main (argc=<optimized out>, argv=<optimized out>) at main.c:1342 e = <optimized out> ltmp = 0xfe5a50 ret = <optimized out> flags = <optimized out> p = <optimized out> worker_pool = <optimized out> main_pool = 0xfe5920 s = 0xfe59e0 creds = {xcred = 0xfe9f10, cprio = 0xfea6e0, dh_params = 0x0} #B Disconnection worker gdb: gdb /usr/bin/ocserv 15964 GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... /usr/bin/ocserv: No such file or directory. Attaching to process 15964 Reading symbols from /usr/local/sbin/ocserv...done. Reading symbols from /usr/lib/x86_64-linux-gnu/libopts.so.25...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libopts.so.25 Reading symbols from /usr/local/lib/libgnutls.so.28...done. Loaded symbols for /usr/local/lib/libgnutls.so.28 Reading symbols from /lib/x86_64-linux-gnu/libpam.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libpam.so.0 Reading symbols from /usr/lib/x86_64-linux-gnu/libseccomp.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libseccomp.so.2 Reading symbols from /lib/x86_64-linux-gnu/libwrap.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libwrap.so.0 Reading symbols from /lib/x86_64-linux-gnu/libcrypt.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libcrypt-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libcrypt.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libhttp_parser.so.2.1 Reading symbols from /lib/libfreeradius-client.so.2...done. Loaded symbols for /lib/libfreeradius-client.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libnettle.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libnettle.so.4 Reading symbols from /usr/lib/x86_64-linux-gnu/libev.so.4...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libev.so.4 Reading symbols from /usr/lib/libpcl.so.1...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libpcl.so.1 Reading symbols from /lib/x86_64-linux-gnu/libc.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libc.so.6 Reading symbols from /lib/x86_64-linux-gnu/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libz.so.1 Reading symbols from /usr/lib/x86_64-linux-gnu/libhogweed.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libhogweed.so.2 Reading symbols from /usr/lib/x86_64-linux-gnu/libgmp.so.10...(no debugging symbols found)...done. Loaded symbols for /usr/lib/x86_64-linux-gnu/libgmp.so.10 Reading symbols from /lib/x86_64-linux-gnu/libaudit.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/x86_64-linux-gnu/libaudit.so.1 Reading symbols from /lib/x86_64-linux-gnu/libdl.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libdl-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libdl.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnsl.so.1...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnsl-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnsl.so.1 Reading symbols from /lib64/ld-linux-x86-64.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/ld-2.19.so...done. done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /lib/x86_64-linux-gnu/libm.so.6...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libm-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libm.so.6 Reading symbols from /lib/x86_64-linux-gnu/libnss_compat.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_compat-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_compat.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnss_nis.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_nis-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_nis.so.2 Reading symbols from /lib/x86_64-linux-gnu/libnss_files.so.2...Reading symbols from /usr/lib/debug//lib/x86_64-linux-gnu/libnss_files-2.19.so...done. done. Loaded symbols for /lib/x86_64-linux-gnu/libnss_files.so.2 0x00007fac5291d417 in __libc_writev (fd=0, vector=0x7ffff8899a90, count=1) at ../sysdeps/unix/sysv/linux/writev.c:49 49 ../sysdeps/unix/sysv/linux/writev.c: No such file or directory. (gdb) bt full #0 0x00007fac5291d417 in __libc_writev (fd=0, vector=0x7ffff8899a90, count=1) at ../sysdeps/unix/sysv/linux/writev.c:49 resultvar = 12 result = <optimized out> #1 0x00007fac53ed9c58 in _gnutls_writev (total=58, giovec_cnt=1, giovec=0x7ffff8899a90, session=0x1016010) at gnutls_buffers.c:483 i = <optimized out> is_dtls = false no_writev = 0 fd = 0x0 #2 _gnutls_io_write_flush (session=session at entry=0x1016010) at gnutls_buffers.c:683 msg = {data = 0x0, size = 0} send_buffer = 0x10168f0 ret = <optimized out> sent = 0 tosend = 58 iovec = {{iov_base = 0x10276d0, iov_len = 58}, {iov_base = 0x10, iov_len = 16883320}, {iov_base = 0x1018d60, iov_len = 1647603077}, {iov_base = 0x10276fa, iov_len = 16883288}, { iov_base = 0x10, iov_len = 16883320}, {iov_base = 0x7ffff8899b10, iov_len = 140378103648064}, {iov_base = 0x10276fa, iov_len = 140378103671172}, {iov_base = 0x414a, iov_len = 16878944}, {iov_base = 0x8e1b82e2935c2b50, iov_len = 12934948170159259015}, {iov_base = 0x0, iov_len = 18319672542873517056}, {iov_base = 0x10276dd, iov_len = 16669880}, {iov_base = 0x0, iov_len = 29}, {iov_base = 0x10276dd, iov_len = 16753016}, {iov_base = 0x414a, iov_len = 140378119245405}, {iov_base = 0xfe5cb8, iov_len = 140378119245632}, {iov_base = 0x35, iov_len = 16714}, {iov_base = 0x1016010, iov_len = 16938624}, {iov_base = 0x0, iov_len = 140378122284096}, {iov_base = 0x35, iov_len = 16714}, {iov_base = 0x1016010, iov_len = 140378119177280}, {iov_base = 0x107, iov_len = 16669880}, {iov_base = 0x7fff0000001d, iov_len = 16938717}, {iov_base = 0x0, iov_len = 4}, {iov_base = 0x8, iov_len = 511101108253}, {iov_base = 0xffa178, iov_len = 140737363156075}, {iov_base = 0x100000000, iov_len = 0}, {iov_base = 0x40000000c, iov_len = 140737363156064}, {iov_base = 0xfe5990, iov_len = 140378119163654}, {iov_base = 0x40, iov_len = 16}, {iov_base = 0x5e3c000000000000, iov_len = 139762530976535}, { iov_base = 0x88874e38, iov_len = 1580990464}, {iov_base = 0xffa178, iov_len = 18319672542873517056}} i = 1 cur = <optimized out> #3 0x00007fac53eda00c in _gnutls_io_write_buffered (session=session at entry=0x1016010, bufel=<optimized out>, mflag=mflag at entry=1) at gnutls_buffers.c:639 send_buffer = 0x10168f0 #4 0x00007fac53ed477c in _gnutls_send_tlen_int (session=session at entry=0x1016010, type=type at entry=GNUTLS_APPLICATION_DATA, htype=htype at entry=4294967295, epoch_rel=epoch_rel at entry=70001, _data=_data at entry=0x1016010, data_size=data_size at entry=29, min_pad=min_pad at entry=0, mflags=mflags at entry=1) at gnutls_record.c:551 bufel = <optimized out> cipher_size = <optimized out> retval = 29 ret = <optimized out> send_data_size = 29 data = 0x1016010 "\001" record_params = 0xfe5990 max_send_size = 12 record_state = 0xfe5c88 #5 0x00007fac53ed6a3a in _gnutls_send_int (mflags=1, data_size=29, _data=0x1016010, epoch_rel=70001, htype=4294967295, type=GNUTLS_APPLICATION_DATA, session=0x1016010) at gnutls_record.h:43 No locals. #6 gnutls_record_send (session=0x1016010, data=data at entry=0xffa178, data_size=data_size at entry=29) at gnutls_record.c:1564 No locals. #7 0x0000000000410c4c in cstp_send (ws=ws at entry=0xff9c30, data=data at entry=0xffa178, data_size=data_size at entry=29) at tlslib.c:92 ret = <optimized out> left = 29 p = 0xffa178 "STF\001" #8 0x000000000040be7d in tun_mainloop (tnow=0x7ffff8899e50, ws=0xff9c30) at worker-vpn.c:1245 ret = <optimized out> ---Type <return> to continue, or q <return> to quit--- e = <optimized out> cstp_type = 0 dtls_to_send = {data = 0xffa178 "STF\001", size = 21} l = <optimized out> tls_retry = <optimized out> dtls_type = 0 cstp_to_send = {data = 0xffa178 "STF\001", size = 21} #9 connect_handler (ws=ws at entry=0xff9c30) at worker-vpn.c:1949 req = 0xff9c60 rfds = {fds_bits = {2, 0 <repeats 15 times>}} e = <optimized out> max = 13 ret = <optimized out> t = 2048 p = <optimized out> rnd = 648176375 tv = {tv_sec = 10, tv_nsec = 0} tls_pending = 0 dtls_pending = 0 i = <optimized out> tnow = {tv_sec = 1453019195, tv_nsec = 285061000} ip6 = <optimized out> sl = 4 emptyset = {__val = {0 <repeats 16 times>}} blockset = {__val = {16384, 0 <repeats 15 times>}} #10 0x000000000040d69c in vpn_server (ws=ws at entry=0xff9c30) at worker-vpn.c:571 ret = <optimized out> nparsed = <optimized out> nrecvd = <optimized out> session = 0x1016010 parser = {type = 0 '\000', flags = 0 '\000', state = 17 '\021', header_state = 0 '\000', index = 0 '\000', nread = 0, content_length = 18446744073709551615, http_major = 1, http_minor = 1, status_code = 0, method = 5 '\005', http_errno = 0 '\000', upgrade = 1 '\001', data = 0xff9c30} settings = {on_message_begin = 0x0, on_url = 0x41db60 <http_url_cb>, on_status_complete = 0x0, on_header_field = 0x41dbb0 <http_header_field_cb>, on_header_value = 0x41dcc0 <http_header_value_cb>, on_headers_complete = 0x41ddc0 <http_header_complete_cb>, on_body = 0x41de50 <http_body_cb>, on_message_complete = 0x41de30 <http_message_complete_cb>} fn = <optimized out> requests_left = 15 #11 0x0000000000408b87 in listen_watcher_cb (loop=0x7fac53002900, w=<optimized out>, revents=<optimized out>) at main.c:1061 stype = 0 s = 0xfe59e0 ltmp = <optimized out> ctmp = 0x0 ws = 0xff9c30 fd = 0 ret = <optimized out> cmd_fd = {1, 13} pid = <optimized out> #12 0x00007fac52df8e45 in ev_invoke_pending () from /usr/lib/x86_64-linux-gnu/libev.so.4 No symbol table info available. ---Type <return> to continue, or q <return> to quit--- #13 0x00007fac52dfbfe7 in ev_run () from /usr/lib/x86_64-linux-gnu/libev.so.4 No symbol table info available. #14 0x0000000000406b42 in main (argc=<optimized out>, argv=<optimized out>) at main.c:1342 e = <optimized out> ltmp = 0xfe5a50 ret = <optimized out> flags = <optimized out> p = <optimized out> worker_pool = <optimized out> main_pool = 0xfe5920 s = 0xfe59e0 creds = {xcred = 0xfe9f10, cprio = 0xfea6e0, dh_params = 0x0}
