On 05/15/2013 01:07 PM, Loïc BLOT wrote: > Sorry, but it's an approximation, when there is only few users squid can > run for hours :) I see. This information increases the number of suspects, but the recommended triaging methods remain the same (FWIW, kdump is not one of them). > Squid has got in an infinite loop. Here is a kdump extract of squid it > seems there is no special kernel call: This kdump does not show a infinite loop -- things change as Squid reads and writes data. Was that kdump growing as Squid was inside the infinite loop? Or did it stop grow once Squid entered the loop? Thank you, Alex. > 22904 squid RET write 5235/0x1473 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.341919 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.259000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.344471 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL read(0x97,0x24a30c000,0x4000) > 22904 squid GIO fd 151 read 198 bytes > "\^V\^C\^A\0\M^F\^P\0\0\M^B\0\M^@\^D\^T\M-X\^] \M-c\M^Y\^\\M-n > \M^H\M-d\M^?\M^G"w\M-- > O(5\M^^\M-Llp\M^]\M^Fj\^]\M^?k:{S\^\)\M-c\M-6\^_~\M^Xw\M-g\^C > \M^Ta\M-|\M-kpQ\v\M-@,\M^C\M^P\^S\M^?m > \M^F\M-=3Dtn\^A\M-V)\M^X\^CXa\M-'\M-c\M-e\M-E?\^RK]\M-a\M-4 > \M^]CJ > \^?&A\M-a > %O\M^S*\M^V3\M-J8\M-C\M-"1\^S\M-cc\a\M-bs\M-=3DX\M-[M\^D\M-B\M^D > \M-<O5`g\^E\M-8\M-Qi\M^_\M-F\M-&\M^@\M-WT\bL\^T\^C\^A\0\^A\^A\^V\^C\^A > \0000\M-*B\M-]\M-S\M->,+\240\M^Y\240o\M-Y\ > \M-$\M-D\M-h\M-G\M-W\M-"\M-V\M-N\M-t\M^H\M-N\^O\M-sbU: \M^W\M^Y > \r\M-5)\M-_\^T\M-.\M^?\M^Gq\M-)\M^Bd\M^@\M^]:\M-m\M^U" > 22904 squid RET read 198/0xc6 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.344488 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.257000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.344505 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL write(0x7f,0x24a30c000,0xc6) > 22904 squid GIO fd 127 wrote 198 bytes > "\^V\^C\^A\0\M^F\^P\0\0\M^B\0\M^@\^D\^T\M-X\^] \M-c\M^Y\^\\M-n > \M^H\M-d\M^?\M^G"w\M-- > O(5\M^^\M-Llp\M^]\M^Fj\^]\M^?k:{S\^\)\M-c\M-6\^_~\M^Xw\M-g\^C > \M^Ta\M-|\M-kpQ\v\M-@,\M^C\M^P\^S\M^?m > \M^F\M-=3Dtn\^A\M-V)\M^X\^CXa\M-'\M-c\M-e\M-E?\^RK]\M-a\M-4 > \M^]CJ > \^?&A\M-a > %O\M^S*\M^V3\M-J8\M-C\M-"1\^S\M-cc\a\M-bs\M-=3DX\M-[M\^D\M-B\M^D > \M-<O5`g\^E\M-8\M-Qi\M^_\M-F\M-&\M^@\M-WT\bL\^T\^C\^A\0\^A\^A\^V\^C\^A > \0000\M-*B\M-]\M-S\M->,+\240\M^Y\240o\M-Y\ > \M-$\M-D\M-h\M-G\M-W\M-"\M-V\M-N\M-t\M^H\M-N\^O\M-sbU: \M^W\M^Y > \r\M-5)\M-_\^T\M-.\M^?\M^Gq\M-)\M^Bd\M^@\M^]:\M-m\M^U" > 22904 squid RET write 198/0xc6 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.344528 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.257000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349041 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL read(0x7f,0x24aa67000,0x4000) > 22904 squid GIO fd 127 read 59 bytes > "\^T\^C\^A\0\^A\^A\^V\^C\^A\0000>\M^V\M-XV"\M^A\M-t@\M^X\a\M-r > \M-@\M-?\M-?4\M-"\M-F@)\^R\M-c\f\^C\M-0\^_\M-<$b3\M-=3D"\\Y\M-8\M-rL\M-E > \M-UK%\M^X\M-o\M-Ou\M-Is\b\M-~" > 22904 squid RET read 59/0x3b > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349057 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.252000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349074 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL write(0x97,0x24aa67000,0x3b) > 22904 squid GIO fd 151 wrote 59 bytes > "\^T\^C\^A\0\^A\^A\^V\^C\^A\0000>\M^V\M-XV"\M^A\M-t@\M^X\a\M-r > \M-@\M-?\M-?4\M-"\M-F@)\^R\M-c\f\^C\M-0\^_\M-<$b3\M-=3D"\\Y\M-8\M-rL\M-E > \M-UK%\M^X\M-o\M-Ou\M-Is\b\M-~" > 22904 squid RET write 59/0x3b > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349096 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.252000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349594 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL read(0x97,0x24a30c000,0x4000) > 22904 squid GIO fd 151 read 277 bytes > "\^W\^C\^A\^A\^P\M-u\M^E\M-1\M-eU\M-s>\M-"Z\M-O\^ZBM\M-3\^EC\M-! > \M-.\^ZR\M-\\^R'\M^Q3\M-3\M-vD\M-Z0\0W(\M^JR\M-I\M-gb\M-g\M-t\f\M-(\M-H > \^?\M-i\^C\^DE\M-2\^^\240b\M-F\M-T\M-w\ > \M-6D\M-V\M-s\M-;\M^V\M-?\M^^\^E-d\M-2 w{5\M-0\M-3!\M-g#T\^^Sa > \M-I\M-5\M^U:\M-!\M-i\M^@\M-N\M-`'\M-*\M-)a\^DW\M^[u~\M-q\M-#\M-y > \M^By{\M->M\M^VR\M-*K\M^^\M^N\^P\^Q\^T\M^LJ\M-V\ > \M->\M-N\M-|\^T!\^C\M-d\M-S\M^]\M^_\M-i\^P\M^K\^R\M-$\M-P > \M-i"@~.\^E:\M^H\M^?\M-\\^Ey\M^Q\^[\M-L8u\M-(\M-R\M-E\M-N\M-m<K\M-T6\M-6 > \M-U\M-CGx\M-&\M-+\M-m\M-E\M-Iv \M-*O\M-i\^O\ > \M-bT\M^U\M-t\M-K#9q\M-s.S\M-J\M-'\^V!\M-0n\M-r\M-?\M-h\M-! > \M-o)U\M-V'\M-*W\^N\M^[m\M-!\M^L\M-}\M-[\M-9\^[kS\M^^\M-[8\M-7M\M-,\M^I! > \^W\M^_\v\^[)\M-H\M-%Y\M-Rg\M^A\M-k\M-#:\ > \M-#UI\M-@\M^XtejD\M-( \M-&\M^]\240)\M-[\^V\M-sL \M^A49\M-2%\M^_ > \M-\I\M-!\M-"\faC\M^H/" > 22904 squid RET read 277/0x115 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349611 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.251000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349627 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL write(0x7f,0x24a30c000,0x115) > 22904 squid GIO fd 127 wrote 277 bytes > "\^W\^C\^A\^A\^P\M-u\M^E\M-1\M-eU\M-s>\M-"Z\M-O\^ZBM\M-3\^EC\M-! > \M-.\^ZR\M-\\^R'\M^Q3\M-3\M-vD\M-Z0\0W(\M^JR\M-I\M-gb\M-g\M-t\f\M-(\M-H > \^?\M-i\^C\^DE\M-2\^^\240b\M-F\M-T\M-w\ > \M-6D\M-V\M-s\M-;\M^V\M-?\M^^\^E-d\M-2 w{5\M-0\M-3!\M-g#T\^^Sa > \M-I\M-5\M^U:\M-!\M-i\M^@\M-N\M-`'\M-*\M-)a\^DW\M^[u~\M-q\M-#\M-y > \M^By{\M->M\M^VR\M-*K\M^^\M^N\^P\^Q\^T\M^LJ\M-V\ > \M->\M-N\M-|\^T!\^C\M-d\M-S\M^]\M^_\M-i\^P\M^K\^R\M-$\M-P > \M-i"@~.\^E:\M^H\M^?\M-\\^Ey\M^Q\^[\M-L8u\M-(\M-R\M-E\M-N\M-m<K\M-T6\M-6 > \M-U\M-CGx\M-&\M-+\M-m\M-E\M-Iv \M-*O\M-i\^O\ > \M-bT\M^U\M-t\M-K#9q\M-s.S\M-J\M-'\^V!\M-0n\M-r\M-?\M-h\M-! > \M-o)U\M-V'\M-*W\^N\M^[m\M-!\M^L\M-}\M-[\M-9\^[kS\M^^\M-[8\M-7M\M-,\M^I! > \^W\M^_\v\^[)\M-H\M-%Y\M-Rg\M^A\M-k\M-#:\ > \M-#UI\M-@\M^XtejD\M-( \M-&\M^]\240)\M-[\^V\M-sL \M^A49\M-2%\M^_ > \M-\I\M-!\M-"\faC\M^H/" > 22904 squid RET write 277/0x115 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.349665 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.251000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.351415 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL read(0x7f,0x24aa67000,0x4000) > 22904 squid GIO fd 127 read 261 bytes > "\^W\^C\^A\^A\0B*9\M-t \M-7\M^H\M^R\^A\^Fj8:.\M-\J\M^Kt{Y > \^]{/\240\M-O\M-fg\^SN\M^U\M-9\M-~_V\M-o;\M-L\M^P\M-c\M-(\^Q\f\M-{\M-3 > \M-}\M-z\M^^e}j\b\M^D\M^X\M^]\M^?7\M-TR\M-f*\ > \M-~\M-91\M^Pbn\^B\M^K\M^_o\M^F\M-yd\^PD\^O\M-u\^D\^Nh\M-2%cv > \M^L\M-Fp\M-+\M-*\M-v\M^O\M^D\M-!\M--\M-t\M-DP2\M-m\M-4\M^S\M-[\M^V\M-N > \M-Ma\M^[0H`v\M-]\M^K\M-V\^\p\0\M-2\M-N<\ > \M-}\^UL\M-[\^S\M-[\^]\M-l\M-2\M-b\^?a\M-G)\M-5\M^QOO\M-J\M-Fi, > \^_\^R\M-*,\M^Nd*\M-L\M-<\^S\M-k&\M^[\^Z\M-f\M-e!\M-Wh\M^QVZV-**\M-ZJ > \M^P\^_9?Wk\^Fb\\\M-C\M-C\M-FI\M^P\M-EDu\ > \M-U\M^T\^C\^D\M-a\M-n\M-<\M-I\M^JA\M-I)QD\M^Z\^Y\M^V\M-N\M-)G > \^V\M^Em7i\M-om&\^B\M^D\^F]f@\M^Kn\^R\240M\M-y\M-Pi\M-x+\fx\^P\M-v%m > \M-to,\M^K\M-*\M-*W\M-6\M-a\M^_[=3DC(h\M-=3Dm\ > \M-ub" > 22904 squid RET read 261/0x105 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.351432 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.250000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.351449 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL write(0x97,0x24aa67000,0x105) > 22904 squid GIO fd 151 wrote 261 bytes > "\^W\^C\^A\^A\0B*9\M-t \M-7\M^H\M^R\^A\^Fj8:.\M-\J\M^Kt{Y > \^]{/\240\M-O\M-fg\^SN\M^U\M-9\M-~_V\M-o;\M-L\M^P\M-c\M-(\^Q\f\M-{\M-3 > \M-}\M-z\M^^e}j\b\M^D\M^X\M^]\M^?7\M-TR\M-f*\ > \M-~\M-91\M^Pbn\^B\M^K\M^_o\M^F\M-yd\^PD\^O\M-u\^D\^Nh\M-2%cv > \M^L\M-Fp\M-+\M-*\M-v\M^O\M^D\M-!\M--\M-t\M-DP2\M-m\M-4\M^S\M-[\M^V\M-N > \M-Ma\M^[0H`v\M-]\M^K\M-V\^\p\0\M-2\M-N<\ > \M-}\^UL\M-[\^S\M-[\^]\M-l\M-2\M-b\^?a\M-G)\M-5\M^QOO\M-J\M-Fi, > \^_\^R\M-*,\M^Nd*\M-L\M-<\^S\M-k&\M^[\^Z\M-f\M-e!\M-Wh\M^QVZV-**\M-ZJ > \M^P\^_9?Wk\^Fb\\\M-C\M-C\M-FI\M^P\M-EDu\ > \M-U\M^T\^C\^D\M-a\M-n\M-<\M-I\M^JA\M-I)QD\M^Z\^Y\M^V\M-N\M-)G > \^V\M^Em7i\M-om&\^B\M^D\^F]f@\M^Kn\^R\240M\M-y\M-Pi\M-x+\fx\^P\M-v%m > \M-to,\M^K\M-*\M-*W\M-6\M-a\M^_[=3DC(h\M-=3Dm\ > \M-ub" > 22904 squid RET write 261/0x105 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.351473 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.250000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.351968 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL read(0x97,0x24a30c000,0x4000) > 22904 squid GIO fd 151 read 37 bytes > "\^U\^C\^A\0 e\^U\M-#\M-!\M-Kna\M^YI\M-CN\^EM\M-I\M-2Q\^A > \M-j,>\M-H\M^La\M-[\M^Lk\M-j\M^@\M^E&v\M-;" > 22904 squid RET read 37/0x25 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.351985 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.249000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.352001 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL write(0x7f,0x24a30c000,0x25) > 22904 squid GIO fd 127 wrote 37 bytes > "\^U\^C\^A\0 e\^U\M-#\M-!\M-Kna\M^YI\M-CN\^EM\M-I\M-2Q\^A > \M-j,>\M-H\M^La\M-[\M^Lk\M-j\M^@\M^E&v\M-;" > 22904 squid RET write 37/0x25 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.352023 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x1,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.249000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.353220 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL read(0x97,0x24a30c000,0x4000) > 22904 squid RET read 0 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.353233 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL write(0x134,0x203603000,0x79) > 22904 squid GIO fd 308 wrote 121 bytes > "1368643633.353 26 10.117.50.145 TCP_MISS/200 5555 CONNECT > wfbs7.icrc.trendmicro.com:443 - HIER_DIRECT/23.35.116.42 - > " > 22904 squid RET write 121/0x79 > 22904 squid CALL close(0x97) > 22904 squid RET close 0 > 22904 squid CALL accept(0x135,0x2491fb100,0x24ad04990) > 22904 squid STRU struct sockaddr { AF_INET, 10.117.50.145:55034 } > 22904 squid RET accept 151/0x97 > 22904 squid CALL getsockname(0x97,0x24b216aa0,0x24ad04990) > 22904 squid STRU struct sockaddr { AF_INET, 10.117.1.1:3128 } > 22904 squid RET getsockname 0 > 22904 squid CALL fcntl(0x97,F_GETFD,<invalid>0x9f830b4) > 22904 squid RET fcntl 0 > 22904 squid CALL fcntl(0x97,F_SETFD,FD_CLOEXEC) > 22904 squid RET fcntl 0 > 22904 squid CALL fcntl(0x97,F_GETFL,0x9f830b4) > 22904 squid RET fcntl 6 > 22904 squid CALL fcntl(0x97,F_SETFL,0x6<O_RDWR|O_NONBLOCK>) > 22904 squid RET fcntl 0 > 22904 squid CALL close(0x7f) > 22904 squid RET close 0 > 22904 squid CALL accept(0x135,0x24b3c9140,0x24a985450) > 22904 squid RET accept -1 errno 35 Resource temporarily > unavailable > 22904 squid CALL __sysctl(4.17.0.2.2.1024,0,0x7f7ffffd1c98,0,0) > 22904 squid RET __sysctl 0 > 22904 squid CALL > __sysctl(4.17.0.2.2.1024,0x24af0b000,0x7f7ffffd1c98,0,0) > 22904 squid RET __sysctl 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0x2,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0.248000000 } > 22904 squid RET kevent 1 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.738596 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.738603 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL > kevent(0x6,0x2056be000,0,0xd12e80,0x80,0x7f7ffffd20a0) > 22904 squid STRU struct timespec { 0 } > 22904 squid RET kevent 9 > 22904 squid CALL gettimeofday(0xcdc6e0,0) > 22904 squid STRU struct timeval { 1368643633.738627 } > 22904 squid RET gettimeofday 0 > 22904 squid CALL accept(0x135,0x24b3c9140,0x24ab80c90) > 22904 squid RET accept -1 errno 35 Resource temporarily > unavailable > 22904 squid CALL read(0x2f7,0x249845000,0x4000) > 22904 squid GIO fd 759 read 89 bytes > "\^W\^C\^A\0003\M-`\M-y_@f\M-lG\M-!r\M-m\M^M\M-K\M^B\^Yp>\M-&\M-m > \M-te\M-l\M^C#.\M^Fc-\M-R\M-d\^? \M-Y\M-%p\M^A.\^T~\M-,\M^SW\M^BX > \M^A\M-@\M-Ys\^Y\a\^V\M-I\^W\^C\^A\0\ > \^\k\M-W\M-D\240\M-f\M-}I\M^U\0\M^^Q\^CcF\M-z\M-:\M^@p > \M^V>\M-]Zb\M-Z`\^X\^P\M-+" > 22904 squid RET read 89/0x59 > 22904 squid CALL read(0x1d2,0x24a34e000,0x4000) > 22904 squid GIO fd 466 read 261 bytes >