On Tue, Aug 5, 2014 at 5:59 PM, Thorwald Lundqvist <thorwald@xxxxxxxxxxxxxx> wrote: > Hi! > > I've just experienced a disk crash in my ceph cluster. This seems to > have caused an error in the kernel since every I/O command sent to any > mapped RBD is defered indefinitely, even an hour later, the OSD is > already taken out of the cluster and ceph -s says HEALTH_OK. > > A kernel dump is provided at this pastebin link: http://pastebin.com/WMqmiUsM > > What did I do prior the crash? > > I mapped one clean rbd device at /dev/rbd0 and one rbd device that had > a partition and ext4 filesystem at 20GB each > > I started to dd rbd1 to rbd0 (dd if=/dev/rbd1 of=/dev/rbd0 bs=4M) It > copied a few GB, then deep-scrub started and at that moment one of the > disks failed. and dd got stuck (disk defered). And it still is. > > I think that only a reboot of the server can fix this problem now. > > This is not normal behavior when a disk crashes, right? No, it's not ;) Technically this is not a crash though, it's more of a deadlock - one process ended up blocking forever while holding a crucial lock. See http://tracker.ceph.com/issues/8818. The fix is going to be in 3.17-rc1 so it should catch the last 3.15 stable update. Thanks, Ilya -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
