On Thu, Jan 09, 2020 at 05:18:02PM +0100, Michal Privoznik wrote:
> Dear list,
>
> QEMU gained support for configuring HMAT recently (see
> v4.2.0-415-g9b12dfa03a
> and friends). HMAT stands for Heterogeneous Memory Attribute Table and
> defines
> various attributes to NUMA. Guest OS/app can read these information and fine
> tune optimization. See [1] for more info (esp. links in the transcript).
>
> QEMU defines so called initiator, which is an attribute to a NUMA node and
> if
> specified points to another node that has the best performance to this node.
>
> For instance:
>
> -machine hmat=on \
> -m 2G,slots=2,maxmem=4G \
> -object memory-backend-ram,size=1G,id=m0 \
> -object memory-backend-ram,size=1G,id=m1 \
> -numa node,nodeid=0,memdev=m0 \
> -numa node,nodeid=1,memdev=m1,initiator=0 \
> -smp 2,sockets=2,maxcpus=2 \
> -numa cpu,node-id=0,socket-id=0 \
> -numa cpu,node-id=0,socket-id=1
>
> creates a machine with 2 NUMA nodes, node 0 has CPUs and node 1 has memory
> only
> and it's initiator is node 0 (yes, HMAT allows you to create CPU-less "NUMA"
> nodes). The initiator of node 0 is not specified, but since the node has at
> least one CPU it is initiator to itself (and has to be per specs).
>
> This could be represented by an attribute to our /domain/cpu/numa/cell
> element.
> For instance like this:
>
> <domain>
> <vcpu>2</vcpu>
> <cpu>
> <numa>
> <cell id='0' cpus='0,1' memory='1' unit='GiB'/>
> <cell id='1' memory='1' unit='GiB' initiator='0'/>
> </numa>
> </cpu>
> </domain>
We've gained an 'initiator' attribute on the cell, and 'cpus' is
optional if 'initiator' is present.
Can we have the opposite - nodes with CPUs, but without local memory ?
eg
<cell id='0' cpus='0,1' unit='GiB'/>
> Then, QEMU allows us to control two other important memory attributes:
>
> 1) hmat-lb for Latency and Bandwidth
>
> 2) hmat-cache for cache attributes
>
> For example:
>
> -machine hmat=on \
> -m 2G,slots=2,maxmem=4G \
> -object memory-backend-ram,size=1G,id=m0 \
> -object memory-backend-ram,size=1G,id=m1 \
> -smp 2,sockets=2,maxcpus=2 \
> -numa node,nodeid=0,memdev=m0 \
> -numa node,nodeid=1,memdev=m1,initiator=0 \
> -numa cpu,node-id=0,socket-id=0 \
> -numa cpu,node-id=0,socket-id=1 \
> -numa hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-latency,latency=5
> \
> -numa hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-bandwidth,bandwidth=200M
> \
> -numa hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-latency,latency=10
> \
> -numa hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-bandwidth,bandwidth=100M
> \
> -numa hmat-cache,node-id=0,size=10K,level=1,associativity=direct,policy=write-back,line=8
> \
> -numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8
>
> This extends previous example by defining some latencies and cache
> attributes.
> The node 0 has access latency of 5 ns and bandwidth of 200MB/s and node 1
> has
> access latency of 10ns and bandwidth of only 100MB/s. The memory cache level
> 1
> on both nodes is 10KB, cache line is 8B long with write-back policy and
> direct
> associativity (whatever that means).
This description doesn't match my understanding of the semantics
for these latency options. Your description here is talking about
latency of a single node at a time. I believe these configs
are talking about latency of the *link* between two nodes.
So
-numa hmat-lb,initiator=0,target=0,hierarchy=memory,data-type=access-latency,latency=5
is a local node access latency as src+dst nodes are the same
but
-numa hmat-lb,initiator=0,target=1,hierarchy=memory,data-type=access-latency,latency=10
is a cross-node access latency for the link between node
0 and node 1.
> For better future extensibility I'd express these as separate elements,
> rather
> than attributes to <cell/> element. For instance like this:
>
> <domain>
> <vcpu>2</vcpu>
> <cpu>
> <numa>
> <cell id='0' cpus='0,1' memory='1' unit='GiB'>
> <latencies>
> <latency type='access' value='5'/>
> <bandwidth type='access' unit='MiB' value='200'/>
> </latencies>
> <caches>
> <cache level='1' associativity='direct' policy='write-back'>
> <size unit='KiB' value='10'/>
> <line unit='B' value='8'/>
> </cache>
> </caches>
> </cell>
> <cell id='1' memory='1' unit='GiB' initiator='0'>
> <latencies>
> <latency type='access' value='10'/>
> <bandwidth type='access' unit='MiB' value='100'/>
> </latencies>
> <caches>
> <cache level='1' associativity='direct' policy='write-back'>
> <size unit='KiB' value='10'/>
> <line unit='B' value='8'/>
> </cache>
> </caches>
> </cell>
> </numa>
We shouldn't have <latencies> as a child of the <cell>, because
we need to describe the latencies for the cross-product of all
cells. Putting latency as a child of a cell means we would have
2 possible places to put the same information - either the source
or target node.
The <caches> info is ok as a child of <cell>, though I'd prefer
to cull the extra <caches> wrapper and make <cache> a direct
child - we can still allow <cache> to be listed multiple times
under <cell> without the extra element.
> </cpu>
> </domain>
>
> Thing is, the @hierarchy argument accepts: memory (referring to whole
> memory),
> or first-level|second-level|third-level (referring to side caches for each
> domain). I haven't figured out yet, how to express the levels in XML yet.
>
> The @data-type argument accepts access|read|write (this is expressed by
> @type
> attribute to <latency/> and <bandwidth/> elements). Latency and bandwidth
> can
> be combined with each type: access-latency, read-latency, write-latency,
> access-bandwidth, read-bandwidth, write-bandwidth. And these 6 can then be
> combined with aforementioned @hierarchy, producing 24 combinations (if I
> read
> qemu cmd line specs correctly [2]).
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
