On 8/22/19 3:59 PM, Jason Dillaman wrote: > On Thu, Aug 22, 2019 at 9:23 AM Wido den Hollander <wido@xxxxxxxx> wrote: >> >> Hi, >> >> In a couple of situations I have encountered that Virtual Machines >> running on RBD had a high I/O-wait, nearly 100%, on their vdX (VirtIO) >> or sdX (Virtio-SCSI) devices while they were performing CPU intensive tasks. >> >> These servers would be running a very CPU intensive application while >> *not* doing that many disk I/O. >> >> I however noticed that the I/O-wait of the disk(s) in the VM went up to >> 100%. >> >> This VM is CPU limited by Libvirt by putting that KVM process in it's >> own cgroup with a CPU limitation. >> >> Now, my theory is: >> >> KVM (qemu-kvm) is completely userspace and librbd runs inside qemu-kvm >> as a library. All threads for disk I/O are part of the same PID and thus >> part of that cgroup. >> >> If a process inside the Virtual Machine now starts to consume all CPU >> time there is nothing left for librbd which slows it down. >> >> This then causes a increased I/O-wait inside the Virtual Machine. Even >> though the VM is not performing a lot of disk I/O. The wait of the I/O >> goes up due to this. >> >> >> Is my theory sane? > > Yes, I would say that your theory is sane. Have you looked into > libvirt's cgroup controls for limiting the emulator portion vs the > vCPUs [1]? I'd hope the librbd code and threads should be running in > the emulator cgroup (in a perfect world). > I checked with 'virsh schedinfo X' and this is the output I got: Scheduler : posix cpu_shares : 1000 vcpu_period : 100000 vcpu_quota : -1 emulator_period: 100000 emulator_quota : -1 global_period : 100000 global_quota : -1 iothread_period: 100000 iothread_quota : -1 How can we confirm if the librbd code runs inside the Emulator part? Wido >> Can somebody confirm this? >> >> Thanks, >> >> Wido >> _______________________________________________ >> ceph-users mailing list >> ceph-users@xxxxxxxxxxxxxx >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > [1] https://libvirt.org/cgroups.html > _______________________________________________ ceph-users mailing list ceph-users@xxxxxxxxxxxxxx http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
 |