On Thu, Dec 20, 2018 at 04:25:55PM -0800, Alexander Duyck wrote:
>On Thu, 2018-12-20 at 14:41 -0800, Andrew Morton wrote:
>> Could someone please review this?
>>
>> Thanks.
>>
>> From: Wei Yang <richard.weiyang@xxxxxxxxx>
>> Subject: mm/slub.c: improve performance by skipping checked node in get_any_partial()
>>
>> 1. Background
>>
>> Current slub has three layers:
>>
>> * cpu_slab
>> * percpu_partial
>> * per node partial list
>>
>> Slub allocator tries to get an object from top to bottom. When it
>> can't get an object from the upper two layers, it will search the per
>> node partial list. The is done in get_partial().
>>
>> The abstraction of get_partial() look like this:
>>
>> get_partial()
>> get_partial_node()
>> get_any_partial()
>> for_each_zone_zonelist()
>>
>> The idea behind this is: first try a local node, then try other nodes
>> if caller doesn't specify a node.
>>
>> 2. Room for Improvement
>>
>> When we look one step deeper in get_any_partial(), it tries to get a
>> proper node by for_each_zone_zonelist(), which iterates on the
>> node_zonelists.
>>
>> This behavior would introduce some redundant check on the same node.
>> Because:
>>
>> * the local node is already checked in get_partial_node()
>> * one node may have several zones on node_zonelists
>>
>
>So it seems like there can be a few different behaviors based on
>mempolicy_slab_node() being used to construct the zonelist. Do you
>happen to know what memory policy your test process was running under?
>Also have you tried using any of the other policies to gather data?
>
I have little knowledge about the mempolicy so the test is based on a
"default" configuration.
>> 3. Solution Proposed in Patch
>>
>> We could reduce these redundant check by record the last unsuccessful
>> node and then skip it.
>>
>> 4. Tests & Result
>>
>> After some tests, the result shows this may improve the system a little,
>> especially on a machine with only one node.
>>
>> 4.1 Test Description
>>
>> There are two cases for two system configurations.
>>
>> Test Cases:
>>
>> 1. counter comparison
>> 2. kernel build test
>>
>> System Configuration:
>>
>> 1. One node machine with 4G
>> 2. Four node machine with 8G
>>
>> 4.2 Result for Test 1
>>
>> Test 1: counter comparison
>>
>> This is a test with hacked kernel to record times function
>> get_any_partial() is invoked and times the inner loop iterates. By
>> comparing the ratio of two counters, we get to know how many inner
>> loops we skipped.
>>
>> Here is a snip of the test patch.
>>
>> ---
>> static void *get_any_partial() {
>>
>> get_partial_count++;
>>
>> do {
>> for_each_zone_zonelist() {
>> get_partial_try_count++;
>> }
>> } while();
>>
>> return NULL;
>> }
>> ---
>>
>> The result of (get_partial_count / get_partial_try_count):
>>
>> +----------+----------------+------------+-------------+
>> | | Base | Patched | Improvement|
>> +----------+----------------+------------+-------------+
>> |One Node | 1:3 | 1:0 | - 100% |
>> +----------+----------------+------------+-------------+
>> |Four Nodes| 1:5.8 | 1:2.5 | - 56% |
>> +----------+----------------+------------+-------------+
>>
>> 4.3 Result for Test 2
>>
>> Test 2: kernel build
>>
>> Command used:
>>
>> > time make -j8 bzImage
>>
>> Each version/system configuration combination has four round kernel
>> build tests. Take the average result of real to compare.
>>
>> +----------+----------------+------------+-------------+
>> | | Base | Patched | Improvement|
>> +----------+----------------+------------+-------------+
>> |One Node | 4m41s | 4m32s | - 4.47% |
>> +----------+----------------+------------+-------------+
>> |Four Nodes| 4m45s | 4m39s | - 2.92% |
>> +----------+----------------+------------+-------------+
>>
>> [akpm@xxxxxxxxxxxxxxxxxxxx: rename variable, tweak comment]
>> Link: http://lkml.kernel.org/r/20181120033119.30013-1-richard.weiyang@xxxxxxxxx
>> Signed-off-by: Wei Yang <richard.weiyang@xxxxxxxxx>
>> Cc: Christoph Lameter <cl@xxxxxxxxx>
>> Cc: Pekka Enberg <penberg@xxxxxxxxxx>
>> Cc: David Rientjes <rientjes@xxxxxxxxxx>
>> Cc: Joonsoo Kim <iamjoonsoo.kim@xxxxxxx>
>> Signed-off-by: Andrew Morton <akpm@xxxxxxxxxxxxxxxxxxxx>
>> ---
>>
>> mm/slub.c | 15 +++++++++++++--
>> 1 file changed, 13 insertions(+), 2 deletions(-)
>>
>> --- a/mm/slub.c~mm-slub-improve-performance-by-skipping-checked-node-in-get_any_partial
>> +++ a/mm/slub.c
>> @@ -1877,7 +1877,7 @@ static void *get_partial_node(struct kme
>> * Get a page from somewhere. Search in increasing NUMA distances.
>> */
>> static void *get_any_partial(struct kmem_cache *s, gfp_t flags,
>> - struct kmem_cache_cpu *c)
>> + struct kmem_cache_cpu *c, int exclude_nid)
>> {
>> #ifdef CONFIG_NUMA
>> struct zonelist *zonelist;
>> @@ -1915,6 +1915,9 @@ static void *get_any_partial(struct kmem
>> for_each_zone_zonelist(zone, z, zonelist, high_zoneidx) {
>> struct kmem_cache_node *n;
>>
>> + if (exclude_nid == zone_to_nid(zone))
>> + continue;
>> +
>> n = get_node(s, zone_to_nid(zone));
>>
>> if (n && cpuset_zone_allowed(zone, flags) &&
>> @@ -1931,6 +1934,14 @@ static void *get_any_partial(struct kmem
>> return object;
>> }
>> }
>> + /*
>> + * Failed to get an object from this node, either
>> + * because
>> + * 1. Failure in the above if check
>> + * 2. NULL return from get_partial_node()
>> + * So skip this node next time.
>> + */
>> + exclude_nid = zone_to_nid(zone);
>> }
>> } while (read_mems_allowed_retry(cpuset_mems_cookie));
>> #endif
>
>So this piece gives me some concerns. You are updating the exclude_nid,
>but as a result you are no longer excluding your original nid. So it
>becomes possible that you are going to go back and search your original
>exlcude_nid on the next pass if the zones are interleaved between nodes
>aren't you?
After Michal's change, current zonelist just have node order.
This means once we have a exclude_nid, we will skip all zones on this
node.
>
>Would it perhaps make more sense to instead replace
>for_each_zone_zonelist with for_each_zone_zonelist_nodemask and then
>just mask out any of the failing nodes?
My first version is implemented in this way. While Michal mentioned it
would be heavy on stack and took much time to copy nodemask_t.
>
>> @@ -1955,7 +1966,7 @@ static void *get_partial(struct kmem_cac
>> if (object || node != NUMA_NO_NODE)
>> return object;
>>
>> - return get_any_partial(s, flags, c);
>> + return get_any_partial(s, flags, c, searchnode);
>> }
>>
>> #ifdef CONFIG_PREEMPT
>> _
>>
--
Wei Yang
Help you, Help me
