On Thu, 26 Apr 2007, Adrian Chadd wrote:
On Thu, Apr 26, 2007, Mark Powell wrote:
When the caches restart, read the COSS dir and then when they finish
reading it they die with the same error again. They are both seemingly in
a loop doing this forever now. However, the other cache is still running
happily (perhaps just luck?).
Ok, the 3rd cache failed with the same error, just after sending that.
I'd try to figure out why the relocation is failing. Find the line in
store_io_coss.c which logs that, stick a break statement in gdb and run
squid inside gdb (squid -ND). Then when the thing fails you'll want to
grab some information about the operation:
print *op
print *pr
(gdb) list
1145 if (errflag > 0) {
1146 errno = errflag;
1147 debug(79, 1) ("storeCossCompletePendingReloc: error: %s\n", xstrerror());
1148 } else {
1149 debug(79, 1) ("storeCossCompletePendingReloc: got failure (%d)\n", errflag);
1150 }
1151 } else {
1152 debug(79, 3) ("COSS Pending Relocate: %d -> %d: completed\n", pr->original_filen, pr->new_filen);
1153 coss_stats.read.success++;
1154 }
(gdb) break 1149
Breakpoint 1 at 0x80ed3e1: file coss/store_io_coss.c, line 1149.
(gdb) info breakpoints
Num Type Disp Enb Address What
1 breakpoint keep y 0x080ed3e1 in storeCossCompletePendingReloc at coss/store_io_coss.c:1149
(gdb) run
Starting program: /usr/local/sbin/squid -YND
warning: Unable to get location for thread creation breakpoint: generic error
[New LWP 100198]
[New Thread 0x829d000 (LWP 100151)]
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x829d000 (LWP 100167)]
0x88278c07 in memcpy () from /lib/libc.so.6
(gdb) print *op
No symbol "op" in current context.
(gdb) print *pr
No symbol "pr" in current context.
(gdb) bt
#0 0x88278c07 in memcpy () from /lib/libc.so.6
#1 0x080ed46d in storeCossCompletePendingReloc (fd=21, my_data=0x8d73150, buf=0xfe89038 "", aio_return=266899512, aio_errno=266899512) at coss/store_io_coss.c:1160
#2 0x080e729f in aioCheckCallbacks (SD=0x82bb0b0) at aufs/async_io.c:319
#3 0x080c97b2 in storeDirCallback () at store_dir.c:508
#4 0x0807b710 in comm_select (msec=10) at comm_generic.c:377
#5 0x080a568a in main (argc=2, argv=0xbfbfebb8) at main.c:837
(gdb)
What am I doing wrong with gdb?
This assumes, of course, you can handle a cache hanging in gdb and not restarting..
I assume this is not something to do with using AIO code inside squid
rather than using the VFS_AIO module? I'm not really clear on the
difference that could make? Could you explain or point me at an
explanation?
Cheers.
Adrian
--
Mark Powell - UNIX System Administrator - The University of Salford
Information Services Division, Clifford Whitworth Building,
Salford University, Manchester, M5 4WT, UK.
Tel: +44 161 295 4837 Fax: +44 161 295 5888 www.pgp.com for PGP key
