From: "Daniel P. Berrange" <berrange@xxxxxxxxxx> Describe the new cgroups layout, how to customize placement of guests and what virsh commands are used to access the parameters. Signed-off-by: Daniel P. Berrange <berrange@xxxxxxxxxx> --- docs/cgroups.html.in | 285 +++++++++++++++++++++++++++++++++++++++++++++++++++ docs/sitemap.html.in | 4 + 2 files changed, 289 insertions(+) create mode 100644 docs/cgroups.html.in diff --git a/docs/cgroups.html.in b/docs/cgroups.html.in new file mode 100644 index 0000000..3be0672 --- /dev/null +++ b/docs/cgroups.html.in @@ -0,0 +1,285 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd";> +<html xmlns="http://www.w3.org/1999/xhtml";> + <body> + <h1>Control Groups Resource Management</h1> + + <ul id="toc"></ul> + + <p> + The QEMU and LXC drivers make use of the Linux "Control Groups" facility + for applying resource management to their virtual machines & containers. + </p> + + <h2><a name="requiredControllers">Required controllers</a></h2> + + <p> + The control groups filesystem supports multiple "controllers". By default + the init system (such as systemd) should mount all controllers compiled + into the kernel at <code>/sys/fs/cgroup/$CONTROLLER-NAME</code>. Libvirt + will never attempt to mount any controllers itself, merely detect where + they are mounted. + </p> + + <p> + The QEMU driver is capable of using the <code>cpuset</code>, + <code>cpu</code>, <code>memory</code>, <code>blkio</code> and + <code>devices</code> controllers. None of them are compulsory. + If any controller is not mounted, the resource management APIs + which use it will cease to operate. It is possible to explicitly + turn off use of a controller, even when mounted, via the + <code>/etc/libvirt/qemu.conf</code> configuration file. + </p> + + <p> + The LXC driver is capable of using the <code>cpuset</code>, + <code>cpu</code>, <code>cpuset</code>, <code>freezer</code>, + <code>memory</code>, <code>blkio</code> and <code>devices</code> + controllers. The <code>cpuset</code>, <code>devices</code> + and <code>memory</code> controllers are compulsory. Without + them mounted, no containers can be started. If any of the + other controllers are not mounted, the resource management APIs + which use them will cease to operate. + </p> + + <h2><a name="currentLayout">Current cgroups layout</a></h2> + + <p> + As of libvirt 1.0.5 or later, the cgroups layout created by libvirt has been + simplified, in order to facilitate the setup of resource control policies by + administrators / management applications. The layout is based on the concepts of + "partitions" and "consumers". Each virtual machine or container is a consumer, + and has a corresponding cgroup named <code>$VMNAME.libvirt-{qemu,lxc}</code>. + Each consumer is associated with exactly one partition, which also have a + corresponding cgroup usually named <code>$PARTNAME.partition</code>. The + exceptions to this naming rule are the three top level default partitions, + named <code>/system</code> (for system services), <code>/user</code> (for + user login sessions) and <code>/machine</code> (for virtual machines and + containers). By default every consumer will of course be associated with + the <code>/machine</code> partition. This leads to a hierarchy that looks + like + </p> + + <pre> +$ROOT + | + +- system + | | + | +- libvirtd.service + | + +- machine + | + +- vm1.libvirt-qemu + | | + | +- emulator + | +- vcpu0 + | +- vcpu1 + | + +- vm2.libvirt-qemu + | | + | +- emulator + | +- vcpu0 + | +- vcpu1 + | + +- vm3.libvirt-qemu + | | + | +- emulator + | +- vcpu0 + | +- vcpu1 + | + +- container1.libvirt-lxc + | + +- container2.libvirt-lxc + | + +- container3.libvirt-lxc + </pre> + + <p> + The default cgroups layout ensures that, when there is contention for + CPU time, it is shared equally between system services, user sessions + and virtual machines / containers. This prevents virtual machines from + locking the administrator out of the host, or impacting execution of + system services. ConverselyWhen there is no contention from + system services / user sessions, it is possible for virtual machines + to fully utilize the host CPUs. + </p> + + <h2><a name="customPartiton">Using custom partitions</a></h2> + + <p> + If there is a need to apply resource constraints to groups of + virtual machines or containers, then the single default + partition <code>/machine</code> may not be sufficiently + flexible. The administrator may wish to sub-divide the + default partition, for example into "testing" and "production" + partitions, and then assign each guest to a specific + sub-partition. This is achieved via a small element addition + to the guest domain XML config, just below the main <code>domain</code> + element + </p> + + <pre> + ... + <resource> + <partition>/machine/production</partition> + </resource> + ... + </pre> + + <p> + Libvirt will not auto-create the cgroups directory to back + this partition. In the future, libvirt / virsh will provide + APIs / commands to create custom partitions, but currently + this is left as an exercise for the administrator. For + example, given the XML config above, the admin would need + to create a cgroup named '/machine/production.partition' + </p> + + <pre> +# cd /sys/fs/cgroup +# for i in blkio cpu,cpuacct cpuset devices freezer memory net_cls perf_event + do + mkdir $i/machine/production.partition + done +# for i in cpuset.cpus cpuset.mems + do + cat cpuset/machine/$i > cpuset/machine/production.partition/$i + done +</pre> + + <p> + <strong>Note:</strong> the cgroups directory created as a ".partition" + suffix, but the XML config does not require this suffix. + </p> + + <p> + <strong>Note:</strong> the ability to place guests in custom + partitions is only available with libvirt >= 1.0.5, using + the new cgroup layout. The legacy cgroups layout described + later did not support customization per guest. + </p> + + <h2><a name="resourceAPIs">Resource management APIs/commands</a></h2> + + <p> + Since libvirt aims to provide an API which is portable across + hypervisors, the concept of cgroups is not exposed directly + in the API or XML configuration. It is considered to be an + internal implementation detail. Instead libvirt provides a + set of APIs for applying resource controls, which are then + mapped to corresponding cgroup tunables + </p> + + <h3>Scheduler tuning</h3> + + <p> + Parameters from the "cpu" controller are exposed via the + <code>schedinfo</code> command in virsh. + </p> + + <pre> +# virsh schedinfo demo +Scheduler : posix +cpu_shares : 1024 +vcpu_period : 100000 +vcpu_quota : -1 +emulator_period: 100000 +emulator_quota : -1</pre> + + + <h3>Block I/O tuning</h3> + + <p> + Parameters from the "blkio" controller are exposed via the + <code>bkliotune</code> command in virsh. + </p> + + + <pre> +# virsh blkiotune demo +weight : 500 +device_weight : </pre> + + <h3>Memory tuning</h3> + + <p> + Parameters from the "memory" controller are exposed via the + <code>memtune</code> command in virsh. + </p> + + <pre> +# virsh memtune demo +hard_limit : 580192 +soft_limit : unlimited +swap_hard_limit: unlimited + </pre> + + <h3>Network tuning</h3> + + <p> + The <code>net_cls</code> is not currently used. Instead traffic + filter policies are set directly against individual virtual + network interfaces. + </p> + + <h2><a name="legacyLayout">Legacy cgroups layout</a></h2> + + <p> + Prior to libvirt 1.0.5, the cgroups layout created by libvirt was different + from that described above, and did not allow for administrator customization. + Libvirt used a fixed, 3-level hiearchy <code>libvirt/{qemu,lxc}/$VMNAME</code> + which was rooted at the point in the hiearchy where libvirtd itself was + located. So if libvirtd was placed at <code>/system/libvirtd.service</code> + by systemd, the groups for each virtual machine / container would be located + at <code>/system/libvirtd.service/libvirt/{qemu,lxc}/$VMNAME</code>. In addition + to this, the QEMU drivers further child groups for each vCPU thread and the + emulator thread(s). This leads to a hiearchy that looked like + </p> + + + <pre> +$ROOT + | + +- system + | + +- libvirtd.service + | + +- libvirt + | + +- qemu + | | + | +- vm1 + | | | + | | +- emulator + | | +- vcpu0 + | | +- vcpu1 + | | + | +- vm2 + | | | + | | +- emulator + | | +- vcpu0 + | | +- vcpu1 + | | + | +- vm3 + | | + | +- emulator + | +- vcpu0 + | +- vcpu1 + | + +- lxc + | + +- container1 + | + +- container2 + | + +- container3 + </pre> + + <p> + Although current releases are much improved, historically the use of deep + hiearchies has had a significant negative impact on the kernel scalability. + The legacy libvirt cgroups layout highlighted these problems, to the detriment + of the performance of virtual machines and containers. + </p> + </body> +</html> diff --git a/docs/sitemap.html.in b/docs/sitemap.html.in index 619e4a1..cb7cc5b 100644 --- a/docs/sitemap.html.in +++ b/docs/sitemap.html.in @@ -89,6 +89,10 @@ <span>Ensuring exclusive guest access to disks</span> </li> <li> + <a href="cgroups.html">CGroups</a> + <span>Control groups integration</span> + </li> + <li> <a href="hooks.html">Hooks</a> <span>Hooks for system specific management</span> </li> -- 1.8.1.4 -- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list