On 10/04/2011 12:41 PM, Laine Stump wrote:
On 10/04/2011 12:59 PM, Eric Blake wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=740899 documents that
if qemu uses aio=native for its disks, then it consumes 128 aio
requests per disk. On a host with multiple guests, this can quickly
run out of kernel aio requests with the default aio-max-nr of
65536. Kernel developers have confirmed that there is no up-front
cost to raising this limit (a larger limit merely implies that more
aio requests can be issued in parallel, which in turn will result
in more kernel memory allocation if the system really does use that
many requests). Since the system default limit prevents 256 disks,
which is well within libvirt's current scalability, this patch
installs a file to raise the limit and document it in case a system
administrator has further cause to tune the limit. The install
only works on platforms new enough to source /etc/sysctl.d/*
alongside /etc/sysctl.conf (F14 and RHEL 6).
ACK. This all makes sense and seems fairly thoroughly researched. (Would
be good to actually try it, though)
Oh sure, guilt me into trying it, to make me find my typo (aio-max-size
was a different tunable, valid in RHEL 3 days but now long-gone).
[In case my tone isn't coming across correctly - that translates to
"thank you for saving me from an embarrassing no-op patch"] :)
After squashing this in, I managed to complete installation of a
self-built rpm and validate that /proc/sys/fs/aio-max-nr did indeed
reflect my new value at system bootup. I must confess, though, that I
was unable to test 8k simultaneous disks with qemu using aio on each
disk (my machine isn't that beefy). But at least seeing the procfs
change take effect makes me feel a lot better, so I've now pushed this.
diff --git i/daemon/libvirtd.sysctl w/daemon/libvirtd.sysctl
index 3c70884..275482c 100644
--- i/daemon/libvirtd.sysctl
+++ w/daemon/libvirtd.sysctl
@@ -5,4 +5,4 @@
# 128 * (number of virtual disks on the host)
# Libvirt uses a default of 1M requests to allow 8k disks, with at most
# 64M of kernel memory if all disks hit an aio request at the same time.
-fs.aio-max-size = 1048576
+fs.aio-max-nr = 1048576
--
Eric Blake eblake@xxxxxxxxxx +1-801-349-2682
Libvirt virtualization library http://libvirt.org
--
libvir-list mailing list
libvir-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvir-list