On 4/26/22 1:25 PM, ying.huang@xxxxxxxxx wrote:
On Mon, 2022-04-25 at 16:45 +0530, Jagdish Gediya wrote:
On Sun, Apr 24, 2022 at 11:19:53AM +0800, ying.huang@xxxxxxxxx wrote:
On Sat, 2022-04-23 at 01:25 +0530, Jagdish Gediya wrote:
Some systems(e.g. PowerVM) can have both DRAM(fast memory) only
NUMA node which are N_MEMORY and slow memory(persistent memory)
only NUMA node which are also N_MEMORY. As the current demotion
target finding algorithm works based on N_MEMORY and best distance,
it will choose DRAM only NUMA node as demotion target instead of
persistent memory node on such systems. If DRAM only NUMA node is
filled with demoted pages then at some point new allocations can
start falling to persistent memory, so basically cold pages are in
fast memor (due to demotion) and new pages are in slow memory, this
is why persistent memory nodes should be utilized for demotion and
dram node should be avoided for demotion so that they can be used
for new allocations.
Current implementation can work fine on the system where the memory
only numa nodes are possible only for persistent/slow memory but it
is not suitable for the like of systems mentioned above.
Can you share the NUMA topology information of your machine? And the
demotion order before and after your change?
Whether it's good to use the PMEM nodes as the demotion targets of the
DRAM-only node too?
$ numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 14272 MB
node 0 free: 13392 MB
node 1 cpus:
node 1 size: 2028 MB
node 1 free: 1971 MB
node distances:
node 0 1
0: 10 40
1: 40 10
1) without N_DEMOTION_TARGETS patch series, 1 is demotion target
for 0 even when 1 is DRAM node and there is no demotion targets for 1.
$ cat /sys/bus/nd/devices/dax0.0/target_node
2
$
# cd /sys/bus/dax/drivers/
:/sys/bus/dax/drivers# ls
device_dax kmem
:/sys/bus/dax/drivers# cd device_dax/
:/sys/bus/dax/drivers/device_dax# echo dax0.0 > unbind
:/sys/bus/dax/drivers/device_dax# echo dax0.0 > ../kmem/new_id
:/sys/bus/dax/drivers/device_dax# numactl -H
available: 3 nodes (0-2)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 14272 MB
node 0 free: 13380 MB
node 1 cpus:
node 1 size: 2028 MB
node 1 free: 1961 MB
node 2 cpus:
node 2 size: 0 MB
node 2 free: 0 MB
node distances:
node 0 1 2
0: 10 40 80
1: 40 10 80
2: 80 80 10
This looks like a virtual machine, not a real machine. That's
unfortunate. I am looking forward to a real issue, not a theoritical
possible issue.
This is the source of confusion i guess. A large class of ppc64 systems
are virtualized. The firmware include a hypervisor (PowerVM) and end
user creates guest (aka LPAR) on them. That is the way end user will use
this system. There is no baremetal access on this (There is an openpower
variant, but all new systems built by IBM these days do have PowerVM on
them).
So this is not a theoretical possibility.
-aneesh
