On 10/23/2016 03:24 PM, Theodore Ts'o wrote:
OK, perhaps this is helpful. I reverted my last patch (so I could
repro the problem), and then added the following patch to fio/mutex.c
(see below).
When it hung, I got the following stack trace on the main thread:
(gdb) where
#0 0x00007ff8c018cf2d in nanosleep () at ../sysdeps/unix/syscall-template.S:81
#1 0x00007ff8c01b4fb4 in usleep (useconds=<optimized out>)
at ../sysdeps/unix/sysv/linux/usleep.c:32
#2 0x000000000045b691 in thread_main (data=0x1bc64000) at backend.c:1738
#3 0x000000000045d623 in run_threads (sk_out=sk_out@entry=0x0) at backend.c:2268
#4 0x000000000045d95d in fio_backend (sk_out=sk_out@entry=0x0) at backend.c:2400
#5 0x000000000040cc18 in main (argc=2, argv=0x7fff8446f0d8, envp=<optimized out>) at fio.c:65
.... and it looks like the reason why we can't take the mutex is that
somehow some thread managed to call stat_exit(), which calls
fio_mutex_down() followed by fio_mutex_remove():
(gdb) p (void *) stat_mutex->owner
$4 = (void *) 0x42fd60 <stat_exit+16>
How this could have happened, I have no idea. The only call to
stat_exit() appears to be at the end of fio_backend(), and the main
thread is still in the middle of calling thread_main() which is called
by run_threads().
Do you have any thoughts? Let me know if there's something I can try.
Wow, that is very odd. That should only be possible if the main thread
exits too early, and calls stat_exit() while we still have jobs running.
That should trigger other funkiness too. I have no idea how that is
happening, but it obviously is since your debugging patch showed that
stat_exit() owns the mutex.
But this gives me more to work from, very useul, thanks Ted! I'll pound
on this tomorrow.
--
Jens Axboe
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
