On Wed, Dec 1, 2010 at 10:16 AM, Freddie Cash <fjwcash@xxxxxxxxx> wrote: > Just an update on this. ÂWe made the change over the weekend to enable > "cache=off" for all the VMs, including the libvirt managed ones (turns > out, libvirtd only reads the .xml files at startup); and enabeld KSM > on the host. > > 5 days later, we have only 700 MB of swap used, and 15.2 GB of VM > committed. ÂThis appears to be the steady-state for the host, as it > hasn't changed (cache, free, and buffer change a bit throughout the > day). > > Unfortunately, this has exposed just how horribly unoptimised our > storage array underneath it. Â:( ÂIt's a single 12-drive RAID6, > auto-carved into 2 TB chunks, and then stitched back together via LVM > into a single volume group. ÂWith each VM getting it's own logical > volume. ÂWe have plans over the Christmas break to Âre-do this as a > RAID50 or possible a RAID10 + RAID50. > > Thanks for all the tips and pointers. ÂI'm starting to get all this > figured out and understood. ÂThere's been a lot of changes since > KVM-72. Â:) One final update for the archives. Further research showed that our I/O setup was non-optimal for the RAID controllers were are using (3Ware 9650SE). Modifying the following sysfs variables dropper our I/O wait down to near 0, allowing the controller to keepup with the requests, and eliminating our use of swap (even without using cache=none): (for each block device) block/sda/queue/scheduler = deadline block/sda/queue/nr_requests = 512 block/sda/queue/max_sectors_kb = 128 block/sda/queue/read_ahead_kb = 16384 We're in the process of adding a separate 6-disk RAID10 array for our most I/O-heavy filesystems, which should help even more. So, it looks like our I/O setup was so horrible originally that the controller couldn't keep up, and more RAM in the host was used for buffering, until the host started swapping, and things would grind to a halt. Over a week later, and we're still sitting with 0 bytes swap used in the host, with several 100 MB of free RAM (fluctuates), and I/O usage rarely above 30%. -- Freddie Cash fjwcash@xxxxxxxxx -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
