On Tue, Aug 25, 2009 at 10:25:08PM +0200, Nikola Ciprich wrote: > Hello Christopher, > > thanks a lot vor this overview, it answers a lot of my questions! > May I suggest You put it somewhere on the wiki so it doesn't get > forgotten in the maillist only? I'll rather try to get the worst issues fixed ASAP. > It also rises few new questions though. We have experienced postgresql > database corruptions lately, two times to be exact. First time, I blamed > server crash, but lately (freshly created) database got corrupted for the > second time and there were no crashes since the initialisation. The server > hardware is surely OK. I didn't have much time to look into this > yet, but Your mail just poked me to return to the subject. The situation > is a bit more complex, as there are additional two layers of storage there: > we're using SATA/SAS drives, network-mirrored by DRBD, clustered LVM on top > of those, and finally qemu-kvm using virtio on top of created logical > volumes. So there are plenty of possible culprits, but Your mention of virtio > unsafeness while using cache=writethrough (which is the default for drive > types other then qcow) leads me to suspicion that this might be the reason of > the problem. Databases are sensitive for requests reordering, so I guess > using virtio for postgres storage was quite stupid from me :( > So my question is, could You please advise me a bit on the storage > configuration? virtio performed much better then SCSI, but of course > data integrity is crucial, so would You suggest rather using SCSI? > DRBD doesn't have problem with barriers, clustered LVM SHOULD not > have problems with it, as we're using just striped volumes, but I'll > check it to be sure. So is it safe for me to keep cache=writethrough > for the database volume? I'm pretty sure one of the many laters in your setup will not pass through write barriers, so defintively make sure your write caches are disabled. Also right now virtio is not a good idea for data integrity. The guest side fix for a setup with cache=writethrough or cache=none on block device without volatile disk write cache is however a trivial one line patch I've already submitted. I've attached it below for reference:
Subject: [PATCH] virtio-blk: set QUEUE_ORDERED_DRAIN by default From: Christoph Hellwig <hch@xxxxxx> Currently virtio-blk doesn't set any QUEUE_ORDERED_ flag by default, which means it does not allow filesystems to use barriers. But the typical use case for virtio-blk is to use a backed that uses synchronous I/O, and in that case we can simply set QUEUE_ORDERED_DRAIN to make the block layer drain the request queue around barrier I/O and provide the semantics that the filesystems need. This is what the SCSI disk driver does for disks that have the write cache disabled. With this patch we incorrectly advertise barrier support if someone configure qemu with write back caching. While this displays wrong information in the guest there is nothing that guest could have done even if we rightfully told it that we do not support any barriers. Signed-off-by: Christoph Hellwig <hch@xxxxxx> Index: linux-2.6/drivers/block/virtio_blk.c =================================================================== --- linux-2.6.orig/drivers/block/virtio_blk.c 2009-08-20 17:41:37.019718433 -0300 +++ linux-2.6/drivers/block/virtio_blk.c 2009-08-20 17:45:40.511747922 -0300 @@ -336,9 +336,16 @@ static int __devinit virtblk_probe(struc vblk->disk->driverfs_dev = &vdev->dev; index++; - /* If barriers are supported, tell block layer that queue is ordered */ + /* + * If barriers are supported, tell block layer that queue is ordered. + * + * If no barriers are supported assume the host uses synchronous + * writes and just drain the the queue before and after the barrier. + */ if (virtio_has_feature(vdev, VIRTIO_BLK_F_BARRIER)) blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_TAG, NULL); + else + blk_queue_ordered(vblk->disk->queue, QUEUE_ORDERED_DRAIN, NULL); /* If disk is read-only in the host, the guest should obey */ if (virtio_has_feature(vdev, VIRTIO_BLK_F_RO))
