Josh,
Logs are uploaded to cephdrop with the file name
mikedawson-rbd-qemu-deadlock.
- At about 2013-08-05 19:46 or 47, we hit the issue, traffic went to 0
- At about 2013-08-05 19:53:51, ran a 'virsh screenshot'
Environment is:
- Ceph 0.61.7 (client is co-mingled with three OSDs)
- rbd cache = true and cache=writeback
- qemu 1.4.0 1.4.0+dfsg-1expubuntu4
- Ubuntu Raring with 3.8.0-25-generic
This issue is reproducible in my environment, and I'm willing to run any
wip branch you need. What else can I provide to help?
Thanks,
Mike Dawson
On 8/5/2013 3:48 AM, Stefan Hajnoczi wrote:
On Sun, Aug 04, 2013 at 03:36:52PM +0200, Oliver Francke wrote:
Am 02.08.2013 um 23:47 schrieb Mike Dawson <mike.dawson@xxxxxxxxxxxx>:
We can "un-wedge" the guest by opening a NoVNC session or running a 'virsh screenshot' command. After that, the guest resumes and runs as expected. At that point we can examine the guest. Each time we'll see:
If virsh screenshot works then this confirms that QEMU itself is still
responding. Its main loop cannot be blocked since it was able to
process the screendump command.
This supports Josh's theory that a callback is not being invoked. The
virtio-blk I/O request would be left in a pending state.
Now here is where the behavior varies between configurations:
On a Windows guest with 1 vCPU, you may see the symptom that the guest no
longer responds to ping.
On a Linux guest with multiple vCPUs, you may see the hung task message
from the guest kernel because other vCPUs are still making progress.
Just the vCPU that issued the I/O request and whose task is in
UNINTERRUPTIBLE state would really be stuck.
Basically, the symptoms depend not just on how QEMU is behaving but also
on the guest kernel and how many vCPUs you have configured.
I think this can explain how both problems you are observing, Oliver and
Mike, are a result of the same bug. At least I hope they are :).
Stefan
_______________________________________________
ceph-users mailing list
ceph-users@xxxxxxxxxxxxxx
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
