|
Hi folks I would like to start a discussion about how to support a new cpu feature in libvirt. CAT support is not fully merged into linux kernel yet, the target release is 4.10, and all patches has been merged into
linux tip branch. So there won’t be interface/design changes. ## Background
Intel_rdt_ui [
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/Documentation/x86/intel_rdt_ui.txt?h=x86/cache ]
## What will libvirt do? ### Questions:
a.
VM has it’s own dedicated l3 cache and also can share other l3 cache.
b.
VM can only use the caches which allocated to it.
c.
Has some pre-defined policies and priority for a VM Like COB [1]
### Propose Changes XML domain user interface changes: Option 1: explicit specify cache allocation for a VM 1 work with numa node
Expose how many l3 cache a VM want to reserved and we require that the l3 cache should be bind on some specify cpu socket, just like what we did for numa node. So finally we can calculate on which CPU socket(cell) we need to allocate how may l3cache for a VM. 2. work with vcpu pin Forget numa part, CAT setting should have relationship with cpu core setting, we can apply CAT policy if VM has set cpu pin setting (only VM won’t be schedule to another CPU sockets) Cache allocation on which CPU socket can be calculate as just as 1. We may need to enable both 1 and 2.
There are several policy for cache allocation: Let’ take some examples: For intel e5 v4 2699(Signal socket), there are 55M l3 cache on the chip , the default of L3 schemata is L3:0=ffffff , it represents to use 20 bit to control l3 cache, each bit will
represent 2.75M, which will be the minimal unit on this host. The allocation policy could be 3 policies :
1.
One priority VM: A priority import VM can be allocated a dedicated amount l3 cache (let’s say 2.75 * 4 = 11M) and it can also reach the left 44 M cache which will be shared with other process and
VM on the same host. So that we need to create a new ‘Partition’ n-20371 root@s2600wt:/sys/fs/resctrl# ls cpus info n-20371 schemata tasks Inside of n-20371 directory: root@s2600wt:/sys/fs/resctrl# ls n-20371/ cpus schemata tasks The schemata content will be L3:0=fffff
The tasks content will be the pids of that VM Along we need to change the default schemata of system: root@s2600wt:/sys/fs/resctrl# cat schemata L3:0=ffff # which can not use the highest 4 bits, only tasks in n-20371 can reach that. In this design , we can only get 1 priority VM. Let’s change it a bit to have 2 VMs The schemata of the 2 VMs could be:
1.
L3:0=ffff0 # could not use the 4 low bits 11M l3 cache
2.
L3:0=0ffff # could not use the 4 high bits 11M l3 cache Default schemata changes to L3:0=0fff0 # default system process could only use the middle 33M l3 cache
2.
Isolated l3 cache dedicated allocation for each VM(if required) A VM can only use the cache allocated to it. For example VM 1 requires to allocate 11 M
It’s schemata will be L3:0=f0000 #
VM 2 requires to allocate 11M It’s schemata will be L3:0=f000 And default schemata will be L3:0=fff In this case, we can create multiple VMs which each of them can have dedicated l3 cache. The disadvantage is that we the allocated cache could be not be shared efficiently.
3.
Isolated l3 cache shared allocation for each VM(if required by user) In this case, we will put some VMs (which consider as noisy neighbors) to a ‘Partition’, restrict them to use the only caches allocated to them, by do this, other much more priority
VM can be ensure to have enough l3 cache Then we should decide how much cache the noisy group should have, and put all of their pids in that tasks file.
Option 2: set cache priority and apply policies Don’t specify cache amount at all, only define cache usage priority when define a VM domain XML. Cache priority will decide how much the VM can use l3 cache on a host, it’s not a quantized. So user don’t need to think about how much cache it should have when define a domain
XML. Libvirt will decide cache allocation by the priority of VM defined and policies using. Disadvantage is that caches ability on different host may be different. Same VM domain XML on different host may have vary caches allocation amount. # Support CAT in libvirt itself or leverage other software COB COB is Intel Cache Orchestrator Balancer (COB). please refer
http://clrgitlab.intel.com/rdt/cob/tree/master COB supports some pre-defined policies, it will monitor cpu/cache/cache missing and do cache allocation based on policy using. If COB support monitor some specified process (VM process) and accept priority defined, it will be good to reuse.
At last the question came out:
Reference [1] COB
http://clrgitlab.intel.com/rdt/cob/tree/master [2] CAT intro:
https://software.intel.com/en-us/articles/software-enabling-for-cache-allocation-technology
[3] kernel Intel_rdt_ui [
https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/Documentation/x86/intel_rdt_ui.txt?h=x86/cache ] Best Regards Eli Qiao(乔立勇）OpenStack Core team OTC Intel. -- |
-- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list
