On 03/17/2010 05:24 PM, Chris Webb wrote:
Avi Kivity<avi@xxxxxxxxxx> writes:
On 03/15/2010 10:23 PM, Chris Webb wrote:
Wasteful duplication of page cache between guest and host notwithstanding,
turning on cache=writeback is a spectacular performance win for our guests.
Is this with qcow2, raw file, or direct volume access?
This is with direct access to logical volumes. No file systems or qcow2 in
the stack. Our typical host has a couple of SATA disks, combined in md
RAID1, chopped up into volumes with LVM2 (really just dm linear targets).
The performance measured outside qemu is excellent, inside qemu-kvm is fine
too until multiple guests are trying to access their drives at once, but
then everything starts to grind badly.
OK.
I can understand it for qcow2, but for direct volume access this
shouldn't happen. The guest schedules as many writes as it can,
followed by a sync. The host (and disk) can then reschedule them
whether they are in the writeback cache or in the block layer, and
must sync in the same way once completed.
I don't really understand what's going on here, but I wonder if the
underlying problem might be that all the O_DIRECT/O_SYNC writes from the
guests go down into the same block device at the bottom of the device mapper
stack, and thus can't be reordered with respect to one another.
They should be reorderable. Otherwise host filesystems on several
volumes would suffer the same problems.
Whether the filesystem is in the host or guest shouldn't matter.
For our
purposes,
Guest AA Guest BB Guest AA Guest BB Guest AA Guest BB
write A1 write A1 write B1
write B1 write A2 write A1
write A2 write B1 write A2
are all equivalent, but the system isn't allowed to reorder in this way
because there isn't a separate request queue for each logical volume, just
the one at the bottom. (I don't know whether nested request queues would
behave remotely reasonably either, though!)
Also, if my guest kernel issues (say) three small writes, one at the start
of the disk, one in the middle, one at the end, and then does a flush, can
virtio really express this as one non-contiguous O_DIRECT write (the three
components of which can be reordered by the elevator with respect to one
another) rather than three distinct O_DIRECT writes which can't be permuted?
Can qemu issue a write like that? cache=writeback + flush allows this to be
optimised by the block layer in the normal way.
Guest side virtio will send this as three requests followed by a flush.
Qemu will issue these as three distinct requests and then flush. The
requests are marked, as Christoph says, in a way that limits their
reorderability, and perhaps if we fix these two problems performance
will improve.
Something that comes to mind is merging of flush requests. If N guests
issue one write and one flush each, we should issue N writes and just
one flush - a flush for the disk applies to all volumes on that disk.
--
error compiling committee.c: too many arguments to function
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxxx For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>