Hi Mike, These are the gdb bt of the core dumps that I got on other servers showing the same behavior. It seems that are all failing at the strcmp. Any ideas what could be causing this? Server 1: gdb /usr/sbin/squid core.2582 GNU gdb Fedora (6.8-24.fc9) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... (no debugging symbols found) warning: Can't read pathname for load map: Input/output error. Reading symbols from /usr/lib64/libtcmalloc.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libtcmalloc.so.0 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libssl.so.7... (no debugging symbols found)...done. Loaded symbols for /lib64/libssl.so.7 Reading symbols from /lib64/libcrypto.so.7...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypto.so.7 Reading symbols from /lib64/libdl.so.2... (no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/librt.so.1... (no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libm.so.6... (no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /lib64/libc.so.6... (no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /usr/lib64/libunwind.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libunwind.so.7 Reading symbols from /usr/lib64/libstdc++.so.6... (no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2... (no debugging symbols found)...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2... (no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /lib64/ld-linux-x86-64.so.2... (no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1... (no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1... (no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 (no debugging symbols found) Core was generated by `(squid) -D'. Program terminated with signal 6, Aborted. [New process 2582] [New process 2613] [New process 2601] [New process 2612] [New process 2615] [New process 2606] [New process 2607] [New process 2605] [New process 2610] [New process 2604] [New process 2614] [New process 2603] [New process 2608] [New process 2602] [New process 2609] [New process 2599] [New process 2611] [New process 2598] [New process 2600] [New process 2597] [New process 2596] [New process 2593] [New process 2595] [New process 2592] [New process 2594] [New process 2590] [New process 2591] Missing separate debuginfos, use: debuginfo-install squid.x86_64 ---Type <return> to continue, or q <return> to quit--- #0 0x0000003931832215 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003931832215 in raise () from /lib64/libc.so.6 #1 0x0000003931833d83 in abort () from /lib64/libc.so.6 #2 0x00000000004993ca in death () #3 <signal handler called> #4 0x00000039318809c2 in strcmp () from /lib64/libc.so.6 #5 0x000000000043270e in ?? () #6 0x00000000004905b4 in ?? () #7 0x000000000049111b in ?? () #8 0x00000000004a69d1 in ?? () #9 0x00000000004a8fca in aioCheckCallbacks () #10 0x0000000000493137 in ?? () #11 0x0000000000438b0e in ?? () #12 0x0000000000469d6d in ?? () #13 0x000000393181e32a in __libc_start_main () from /lib64/libc.so.6 #14 0x0000000000409cb9 in drand48 () #15 0x00007fff93828708 in ?? () #16 0x000000000000001c in ?? () #17 0x0000000000000002 in ?? () #18 0x00007fff93828e92 in ?? () #19 0x00007fff93828e9a in ?? () #20 0x0000000000000000 in ?? () Server 2 gdb /usr/sbin/squid core.2591 GNU gdb Fedora (6.8-24.fc9) Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu"... (no debugging symbols found) warning: Can't read pathname for load map: Input/output error. Reading symbols from /usr/lib64/libtcmalloc.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libtcmalloc.so.0 Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypt.so.1 Reading symbols from /lib64/libssl.so.7... (no debugging symbols found)...done. Loaded symbols for /lib64/libssl.so.7 Reading symbols from /lib64/libcrypto.so.7...(no debugging symbols found)...done. Loaded symbols for /lib64/libcrypto.so.7 Reading symbols from /lib64/libdl.so.2... (no debugging symbols found)...done. Loaded symbols for /lib64/libdl.so.2 Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libz.so.1 Reading symbols from /lib64/librt.so.1... (no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/libm.so.6... (no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /lib64/libc.so.6... (no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /usr/lib64/libunwind.so.7...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libunwind.so.7 Reading symbols from /usr/lib64/libstdc++.so.6... (no debugging symbols found)...done. Loaded symbols for /usr/lib64/libstdc++.so.6 Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/libgcc_s.so.1 Reading symbols from /usr/lib64/libgssapi_krb5.so.2... (no debugging symbols found)...done. Loaded symbols for /usr/lib64/libgssapi_krb5.so.2 Reading symbols from /usr/lib64/libkrb5.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5.so.3 Reading symbols from /lib64/libcom_err.so.2... (no debugging symbols found)...done. Loaded symbols for /lib64/libcom_err.so.2 Reading symbols from /usr/lib64/libk5crypto.so.3...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libk5crypto.so.3 Reading symbols from /lib64/ld-linux-x86-64.so.2... (no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 Reading symbols from /usr/lib64/libkrb5support.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib64/libkrb5support.so.0 Reading symbols from /lib64/libkeyutils.so.1... (no debugging symbols found)...done. Loaded symbols for /lib64/libkeyutils.so.1 Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libresolv.so.2 Reading symbols from /lib64/libselinux.so.1... (no debugging symbols found)...done. Loaded symbols for /lib64/libselinux.so.1 Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/libnss_files.so.2 (no debugging symbols found) Core was generated by `(squid) -D'. Program terminated with signal 6, Aborted. [New process 2591] [New process 2609] [New process 2624] [New process 2616] [New process 2622] [New process 2621] [New process 2618] [New process 2620] [New process 2614] [New process 2623] [New process 2615] [New process 2607] [New process 2613] [New process 2611] [New process 2610] [New process 2606] [New process 2605] [New process 2601] [New process 2619] [New process 2617] [New process 2612] [New process 2608] [New process 2604] [New process 2603] [New process 2602] [New process 2600] [New process 2599] Missing separate debuginfos, use: debuginfo-install squid.x86_64 ---Type <return> to continue, or q <return> to quit--- #0 0x0000003931832215 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x0000003931832215 in raise () from /lib64/libc.so.6 #1 0x0000003931833d83 in abort () from /lib64/libc.so.6 #2 0x00000000004993ca in death () #3 <signal handler called> #4 0x00000039318809c2 in strcmp () from /lib64/libc.so.6 #5 0x000000000043270e in ?? () #6 0x00000000004905b4 in ?? () #7 0x000000000049111b in ?? () #8 0x00000000004a69d1 in ?? () #9 0x00000000004a8fca in aioCheckCallbacks () #10 0x0000000000493137 in ?? () #11 0x0000000000438b0e in ?? () #12 0x0000000000469d6d in ?? () #13 0x000000393181e32a in __libc_start_main () from /lib64/libc.so.6 #14 0x0000000000409cb9 in drand48 () #15 0x00007fffdec70c78 in ?? () #16 0x000000000000001c in ?? () #17 0x0000000000000002 in ?? () #18 0x00007fffdec70e92 in ?? () #19 0x00007fffdec70e9a in ?? () #20 0x0000000000000000 in ?? () (gdb) Ragheb Rustom Smartelecom S.A.R.L Sin el fil - Mar Elias Street Absi Center - Ground Floor Beirut, Lebanon Telefax: +961-1-487275 Mobile: +961-3-286282 Email: ragheb@xxxxxxxxxxxxxxx -----Original Message----- From: Mike Marchywka [mailto:marchywka@xxxxxxxxxxx] Sent: Monday, November 30, 2009 1:50 AM To: gkinkie@xxxxxxxxx; ragheb@xxxxxxxxxxxxxxx Cc: squid-users@xxxxxxxxxxxxxxx Subject: RE: squid crashes by itself and reboots automatically ---------------------------------------- > Date: Mon, 30 Nov 2009 00:24:38 +0100 > From: gkinkie@xxxxxxxxx > To: ragheb@xxxxxxxxxxxxxxx > CC: squid-users@xxxxxxxxxxxxxxx > Subject: Re: squid crashes by itself and reboots automatically > > On Sun, Nov 29, 2009 at 8:25 PM, Ragheb Rustom wrote: >> Dear all, >> >> I have multiple servers running squid 2.7-stable7. Lately I have noticed >> that most of these servers are crashing sometimes everyday creating a core >> dump file and then after creating the file restart by itself. After reading >> the core file with gdb and doing a backtrace I got the following info. Can >> anyone please help me identify what is going on with these squid servers. >> You help is very valuable. > > Is there anything in cache.log? Possibly just the few lines before the restart. the default config file seems good about logging stuff. Was that sparse stack trace typical or just the first one you had? The only sig that was obvious was strcmp- if it was blowing up there you either had pathological strings or memory corruption or timing issue etc. Not too informative until you at least know if it is a consistent failure point. > > > -- > /kinkie _________________________________________________________________ Windows 7: I wanted simpler, now it's simpler. I'm a rock star. http://www.microsoft.com/Windows/windows-7/default.aspx?h=myidea?ocid=PID247 27::T:WLMTAGL:ON:WL:en-US:WWL_WIN_myidea:112009
