On 05/22/2012 11:07 PM, Tejun Heo wrote:
Hello, Asias.
On Tue, May 22, 2012 at 02:48:55PM +0800, Asias He wrote:
I actually saw this happened though it should not happen. I have no
idea why this happens. Maybe unbalanced prepare_to_wait_exclusive()
in get_request_wait() and wake_up() in __freed_request()?
Hmm.... unbalanced how? I think bugs in this area are much more
likely to show up as live queue hang rather than issues during queue
shutdown.
I added some debug code to count the number of sleep and wakeup in
get_request_wait() and __freed_request().
I found this after queue cleanup. rl->wait[] is not empty while
rl->count[] == 0. There are exactly nr_sleep - nr_wakeup of process in D
state. So missed wakeup happens? Any ideas to do more debug to find the
root-cause?
[ 52.917115] ---> nr_sleep=1046, nr_wakeup=873, delta=173
$ vmstat 1
1 173 0 712640 24292 96172 0 0 0 0 419 757 0 0 0 100 0
0 173 0 712764 24292 96180 0 0 0 0 472 725 0 1 0 97 2
With this happened, I saw some fio threads in D state which are
sleeping on get_request_wait(). If I wake up the threads in the wait
queue in q->abort_queue_fn() callback which i proposed in the 1/5 of
this patch set, the queue cleanup and thus hot-unplug went pretty
well. (Passed 3000~ rounds of test, without this 2~ round of test
would fail). See this patch [RFC PATCH 4/5] virtio-blk: Use
q->abort_queue_fn() to abort.
If the problem is that easily reproducible (you mean that you can
reproduce it every other time, right?), it would be immensely helpful
if you can root cause the issue properly. As it currently stands,
this series seems to work around the problem by adding extra API
without properly root-causing it. Workarounds without proper
root-causing are already pretty bad and adding extra API for that is
rather silly, IMHO.
Yes. it is very easy to reproduce. /me Trying to figure the root cause out.
--
Asias
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
