Re: [PATCH v2 3/3] mm/page_alloc: Introduce a new sysctl knob vm.pcp_batch_scale_max

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Yafang Shao <laoar.shao@xxxxxxxxx> writes:

> On Mon, Jul 29, 2024 at 2:04 PM Huang, Ying <ying.huang@xxxxxxxxx> wrote:
>>
>> Yafang Shao <laoar.shao@xxxxxxxxx> writes:
>>
>> > On Mon, Jul 29, 2024 at 1:54 PM Huang, Ying <ying.huang@xxxxxxxxx> wrote:
>> >>
>> >> Yafang Shao <laoar.shao@xxxxxxxxx> writes:
>> >>
>> >> > On Mon, Jul 29, 2024 at 1:16 PM Huang, Ying <ying.huang@xxxxxxxxx> wrote:
>> >> >>
>> >> >> Yafang Shao <laoar.shao@xxxxxxxxx> writes:
>> >> >>
>> >> >> > On Mon, Jul 29, 2024 at 11:22 AM Huang, Ying <ying.huang@xxxxxxxxx> wrote:
>> >> >> >>
>> >> >> >> Hi, Yafang,
>> >> >> >>
>> >> >> >> Yafang Shao <laoar.shao@xxxxxxxxx> writes:
>> >> >> >>
>> >> >> >> > During my recent work to resolve latency spikes caused by zone->lock
>> >> >> >> > contention[0], I found that CONFIG_PCP_BATCH_SCALE_MAX is difficult to use
>> >> >> >> > in practice.
>> >> >> >>
>> >> >> >> As we discussed before [1], I still feel confusing about the description
>> >> >> >> about zone->lock contention.  How about change the description to
>> >> >> >> something like,
>> >> >> >
>> >> >> > Sure, I will change it.
>> >> >> >
>> >> >> >>
>> >> >> >> Larger page allocation/freeing batch number may cause longer run time of
>> >> >> >> code holding zone->lock.  If zone->lock is heavily contended at the same
>> >> >> >> time, latency spikes may occur even for casual page allocation/freeing.
>> >> >> >> Although reducing the batch number cannot make zone->lock contended
>> >> >> >> lighter, it can reduce the latency spikes effectively.
>> >> >> >>
>> >> >> >> [1] https://lore.kernel.org/linux-mm/87ttgv8hlz.fsf@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/
>> >> >> >>
>> >> >> >> > To demonstrate this, I wrote a Python script:
>> >> >> >> >
>> >> >> >> >   import mmap
>> >> >> >> >
>> >> >> >> >   size = 6 * 1024**3
>> >> >> >> >
>> >> >> >> >   while True:
>> >> >> >> >       mm = mmap.mmap(-1, size)
>> >> >> >> >       mm[:] = b'\xff' * size
>> >> >> >> >       mm.close()
>> >> >> >> >
>> >> >> >> > Run this script 10 times in parallel and measure the allocation latency by
>> >> >> >> > measuring the duration of rmqueue_bulk() with the BCC tools
>> >> >> >> > funclatency[1]:
>> >> >> >> >
>> >> >> >> >   funclatency -T -i 600 rmqueue_bulk
>> >> >> >> >
>> >> >> >> > Here are the results for both AMD and Intel CPUs.
>> >> >> >> >
>> >> >> >> > AMD EPYC 7W83 64-Core Processor, single NUMA node, KVM virtual server
>> >> >> >> > =====================================================================
>> >> >> >> >
>> >> >> >> > - Default value of 5
>> >> >> >> >
>> >> >> >> >      nsecs               : count     distribution
>> >> >> >> >          0 -> 1          : 0        |                                        |
>> >> >> >> >          2 -> 3          : 0        |                                        |
>> >> >> >> >          4 -> 7          : 0        |                                        |
>> >> >> >> >          8 -> 15         : 0        |                                        |
>> >> >> >> >         16 -> 31         : 0        |                                        |
>> >> >> >> >         32 -> 63         : 0        |                                        |
>> >> >> >> >         64 -> 127        : 0        |                                        |
>> >> >> >> >        128 -> 255        : 0        |                                        |
>> >> >> >> >        256 -> 511        : 0        |                                        |
>> >> >> >> >        512 -> 1023       : 12       |                                        |
>> >> >> >> >       1024 -> 2047       : 9116     |                                        |
>> >> >> >> >       2048 -> 4095       : 2004     |                                        |
>> >> >> >> >       4096 -> 8191       : 2497     |                                        |
>> >> >> >> >       8192 -> 16383      : 2127     |                                        |
>> >> >> >> >      16384 -> 32767      : 2483     |                                        |
>> >> >> >> >      32768 -> 65535      : 10102    |                                        |
>> >> >> >> >      65536 -> 131071     : 212730   |*******************                     |
>> >> >> >> >     131072 -> 262143     : 314692   |*****************************           |
>> >> >> >> >     262144 -> 524287     : 430058   |****************************************|
>> >> >> >> >     524288 -> 1048575    : 224032   |********************                    |
>> >> >> >> >    1048576 -> 2097151    : 73567    |******                                  |
>> >> >> >> >    2097152 -> 4194303    : 17079    |*                                       |
>> >> >> >> >    4194304 -> 8388607    : 3900     |                                        |
>> >> >> >> >    8388608 -> 16777215   : 750      |                                        |
>> >> >> >> >   16777216 -> 33554431   : 88       |                                        |
>> >> >> >> >   33554432 -> 67108863   : 2        |                                        |
>> >> >> >> >
>> >> >> >> > avg = 449775 nsecs, total: 587066511229 nsecs, count: 1305242
>> >> >> >> >
>> >> >> >> > The avg alloc latency can be 449us, and the max latency can be higher
>> >> >> >> > than 30ms.
>> >> >> >> >
>> >> >> >> > - Value set to 0
>> >> >> >> >
>> >> >> >> >      nsecs               : count     distribution
>> >> >> >> >          0 -> 1          : 0        |                                        |
>> >> >> >> >          2 -> 3          : 0        |                                        |
>> >> >> >> >          4 -> 7          : 0        |                                        |
>> >> >> >> >          8 -> 15         : 0        |                                        |
>> >> >> >> >         16 -> 31         : 0        |                                        |
>> >> >> >> >         32 -> 63         : 0        |                                        |
>> >> >> >> >         64 -> 127        : 0        |                                        |
>> >> >> >> >        128 -> 255        : 0        |                                        |
>> >> >> >> >        256 -> 511        : 0        |                                        |
>> >> >> >> >        512 -> 1023       : 92       |                                        |
>> >> >> >> >       1024 -> 2047       : 8594     |                                        |
>> >> >> >> >       2048 -> 4095       : 2042818  |******                                  |
>> >> >> >> >       4096 -> 8191       : 8737624  |**************************              |
>> >> >> >> >       8192 -> 16383      : 13147872 |****************************************|
>> >> >> >> >      16384 -> 32767      : 8799951  |**************************              |
>> >> >> >> >      32768 -> 65535      : 2879715  |********                                |
>> >> >> >> >      65536 -> 131071     : 659600   |**                                      |
>> >> >> >> >     131072 -> 262143     : 204004   |                                        |
>> >> >> >> >     262144 -> 524287     : 78246    |                                        |
>> >> >> >> >     524288 -> 1048575    : 30800    |                                        |
>> >> >> >> >    1048576 -> 2097151    : 12251    |                                        |
>> >> >> >> >    2097152 -> 4194303    : 2950     |                                        |
>> >> >> >> >    4194304 -> 8388607    : 78       |                                        |
>> >> >> >> >
>> >> >> >> > avg = 19359 nsecs, total: 708638369918 nsecs, count: 36604636
>> >> >> >> >
>> >> >> >> > The avg was reduced significantly to 19us, and the max latency is reduced
>> >> >> >> > to less than 8ms.
>> >> >> >> >
>> >> >> >> > - Conclusion
>> >> >> >> >
>> >> >> >> > On this AMD CPU, reducing vm.pcp_batch_scale_max significantly helps reduce
>> >> >> >> > latency. Latency-sensitive applications will benefit from this tuning.
>> >> >> >> >
>> >> >> >> > However, I don't have access to other types of AMD CPUs, so I was unable to
>> >> >> >> > test it on different AMD models.
>> >> >> >> >
>> >> >> >> > Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz, two NUMA nodes
>> >> >> >> > ============================================================
>> >> >> >> >
>> >> >> >> > - Default value of 5
>> >> >> >> >
>> >> >> >> >      nsecs               : count     distribution
>> >> >> >> >          0 -> 1          : 0        |                                        |
>> >> >> >> >          2 -> 3          : 0        |                                        |
>> >> >> >> >          4 -> 7          : 0        |                                        |
>> >> >> >> >          8 -> 15         : 0        |                                        |
>> >> >> >> >         16 -> 31         : 0        |                                        |
>> >> >> >> >         32 -> 63         : 0        |                                        |
>> >> >> >> >         64 -> 127        : 0        |                                        |
>> >> >> >> >        128 -> 255        : 0        |                                        |
>> >> >> >> >        256 -> 511        : 0        |                                        |
>> >> >> >> >        512 -> 1023       : 2419     |                                        |
>> >> >> >> >       1024 -> 2047       : 34499    |*                                       |
>> >> >> >> >       2048 -> 4095       : 4272     |                                        |
>> >> >> >> >       4096 -> 8191       : 9035     |                                        |
>> >> >> >> >       8192 -> 16383      : 4374     |                                        |
>> >> >> >> >      16384 -> 32767      : 2963     |                                        |
>> >> >> >> >      32768 -> 65535      : 6407     |                                        |
>> >> >> >> >      65536 -> 131071     : 884806   |****************************************|
>> >> >> >> >     131072 -> 262143     : 145931   |******                                  |
>> >> >> >> >     262144 -> 524287     : 13406    |                                        |
>> >> >> >> >     524288 -> 1048575    : 1874     |                                        |
>> >> >> >> >    1048576 -> 2097151    : 249      |                                        |
>> >> >> >> >    2097152 -> 4194303    : 28       |                                        |
>> >> >> >> >
>> >> >> >> > avg = 96173 nsecs, total: 106778157925 nsecs, count: 1110263
>> >> >> >> >
>> >> >> >> > - Conclusion
>> >> >> >> >
>> >> >> >> > This Intel CPU works fine with the default setting.
>> >> >> >> >
>> >> >> >> > Intel(R) Xeon(R) Platinum 8260 CPU @ 2.40GHz, single NUMA node
>> >> >> >> > ==============================================================
>> >> >> >> >
>> >> >> >> > Using the cpuset cgroup, we can restrict the test script to run on NUMA
>> >> >> >> > node 0 only.
>> >> >> >> >
>> >> >> >> > - Default value of 5
>> >> >> >> >
>> >> >> >> >      nsecs               : count     distribution
>> >> >> >> >          0 -> 1          : 0        |                                        |
>> >> >> >> >          2 -> 3          : 0        |                                        |
>> >> >> >> >          4 -> 7          : 0        |                                        |
>> >> >> >> >          8 -> 15         : 0        |                                        |
>> >> >> >> >         16 -> 31         : 0        |                                        |
>> >> >> >> >         32 -> 63         : 0        |                                        |
>> >> >> >> >         64 -> 127        : 0        |                                        |
>> >> >> >> >        128 -> 255        : 0        |                                        |
>> >> >> >> >        256 -> 511        : 46       |                                        |
>> >> >> >> >        512 -> 1023       : 695      |                                        |
>> >> >> >> >       1024 -> 2047       : 19950    |*                                       |
>> >> >> >> >       2048 -> 4095       : 1788     |                                        |
>> >> >> >> >       4096 -> 8191       : 3392     |                                        |
>> >> >> >> >       8192 -> 16383      : 2569     |                                        |
>> >> >> >> >      16384 -> 32767      : 2619     |                                        |
>> >> >> >> >      32768 -> 65535      : 3809     |                                        |
>> >> >> >> >      65536 -> 131071     : 616182   |****************************************|
>> >> >> >> >     131072 -> 262143     : 295587   |*******************                     |
>> >> >> >> >     262144 -> 524287     : 75357    |****                                    |
>> >> >> >> >     524288 -> 1048575    : 15471    |*                                       |
>> >> >> >> >    1048576 -> 2097151    : 2939     |                                        |
>> >> >> >> >    2097152 -> 4194303    : 243      |                                        |
>> >> >> >> >    4194304 -> 8388607    : 3        |                                        |
>> >> >> >> >
>> >> >> >> > avg = 144410 nsecs, total: 150281196195 nsecs, count: 1040651
>> >> >> >> >
>> >> >> >> > The zone->lock contention becomes severe when there is only a single NUMA
>> >> >> >> > node. The average latency is approximately 144us, with the maximum
>> >> >> >> > latency exceeding 4ms.
>> >> >> >> >
>> >> >> >> > - Value set to 0
>> >> >> >> >
>> >> >> >> >      nsecs               : count     distribution
>> >> >> >> >          0 -> 1          : 0        |                                        |
>> >> >> >> >          2 -> 3          : 0        |                                        |
>> >> >> >> >          4 -> 7          : 0        |                                        |
>> >> >> >> >          8 -> 15         : 0        |                                        |
>> >> >> >> >         16 -> 31         : 0        |                                        |
>> >> >> >> >         32 -> 63         : 0        |                                        |
>> >> >> >> >         64 -> 127        : 0        |                                        |
>> >> >> >> >        128 -> 255        : 0        |                                        |
>> >> >> >> >        256 -> 511        : 24       |                                        |
>> >> >> >> >        512 -> 1023       : 2686     |                                        |
>> >> >> >> >       1024 -> 2047       : 10246    |                                        |
>> >> >> >> >       2048 -> 4095       : 4061529  |*********                               |
>> >> >> >> >       4096 -> 8191       : 16894971 |****************************************|
>> >> >> >> >       8192 -> 16383      : 6279310  |**************                          |
>> >> >> >> >      16384 -> 32767      : 1658240  |***                                     |
>> >> >> >> >      32768 -> 65535      : 445760   |*                                       |
>> >> >> >> >      65536 -> 131071     : 110817   |                                        |
>> >> >> >> >     131072 -> 262143     : 20279    |                                        |
>> >> >> >> >     262144 -> 524287     : 4176     |                                        |
>> >> >> >> >     524288 -> 1048575    : 436      |                                        |
>> >> >> >> >    1048576 -> 2097151    : 8        |                                        |
>> >> >> >> >    2097152 -> 4194303    : 2        |                                        |
>> >> >> >> >
>> >> >> >> > avg = 8401 nsecs, total: 247739809022 nsecs, count: 29488508
>> >> >> >> >
>> >> >> >> > After setting it to 0, the avg latency is reduced to around 8us, and the
>> >> >> >> > max latency is less than 4ms.
>> >> >> >> >
>> >> >> >> > - Conclusion
>> >> >> >> >
>> >> >> >> > On this Intel CPU, this tuning doesn't help much. Latency-sensitive
>> >> >> >> > applications work well with the default setting.
>> >> >> >> >
>> >> >> >> > It is worth noting that all the above data were tested using the upstream
>> >> >> >> > kernel.
>> >> >> >> >
>> >> >> >> > Why introduce a systl knob?
>> >> >> >> > ===========================
>> >> >> >> >
>> >> >> >> > From the above data, it's clear that different CPU types have varying
>> >> >> >> > allocation latencies concerning zone->lock contention. Typically, people
>> >> >> >> > don't release individual kernel packages for each type of x86_64 CPU.
>> >> >> >> >
>> >> >> >> > Furthermore, for latency-insensitive applications, we can keep the default
>> >> >> >> > setting for better throughput. In our production environment, we set this
>> >> >> >> > value to 0 for applications running on Kubernetes servers while keeping it
>> >> >> >> > at the default value of 5 for other applications like big data. It's not
>> >> >> >> > common to release individual kernel packages for each application.
>> >> >> >>
>> >> >> >> Thanks for detailed performance data!
>> >> >> >>
>> >> >> >> Is there any downside observed to set CONFIG_PCP_BATCH_SCALE_MAX to 0 in
>> >> >> >> your environment?  If not, I suggest to use 0 as default for
>> >> >> >> CONFIG_PCP_BATCH_SCALE_MAX.  Because we have clear evidence that
>> >> >> >> CONFIG_PCP_BATCH_SCALE_MAX hurts latency for some workloads.  After
>> >> >> >> that, if someone found some other workloads need larger
>> >> >> >> CONFIG_PCP_BATCH_SCALE_MAX, we can make it tunable dynamically.
>> >> >> >>
>> >> >> >
>> >> >> > The decision doesn’t rest with us, the kernel team at our company.
>> >> >> > It’s made by the system administrators who manage a large number of
>> >> >> > servers. The latency spikes only occur on the Kubernetes (k8s)
>> >> >> > servers, not in other environments like big data servers. We have
>> >> >> > informed other system administrators, such as those managing the big
>> >> >> > data servers, about the latency spike issues, but they are unwilling
>> >> >> > to make the change.
>> >> >> >
>> >> >> > No one wants to make changes unless there is evidence showing that the
>> >> >> > old settings will negatively impact them. However, as you know,
>> >> >> > latency is not a critical concern for big data; throughput is more
>> >> >> > important. If we keep the current settings, we will have to release
>> >> >> > different kernel packages for different environments, which is a
>> >> >> > significant burden for us.
>> >> >>
>> >> >> Totally understand your requirements.  And, I think that this is better
>> >> >> to be resolved in your downstream kernel.  If there are clear evidences
>> >> >> to prove small batch number hurts throughput for some workloads, we can
>> >> >> make the change in the upstream kernel.
>> >> >>
>> >> >
>> >> > Please don't make this more complicated. We are at an impasse.
>> >> >
>> >> > The key issue here is that the upstream kernel has a default value of
>> >> > 5, not 0. If you can change it to 0, we can persuade our users to
>> >> > follow the upstream changes. They currently set it to 5, not because
>> >> > you, the author, chose this value, but because it is the default in
>> >> > Linus's tree. Since it's in Linus's tree, kernel developers worldwide
>> >> > support it. It's not just your decision as the author, but the entire
>> >> > community supports this default.
>> >> >
>> >> > If, in the future, we find that the value of 0 is not suitable, you'll
>> >> > tell us, "It is an issue in your downstream kernel, not in the
>> >> > upstream kernel, so we won't accept it."  PANIC.
>> >>
>> >> I don't think so.  I suggest you to change the default value to 0.  If
>> >> someone reported that his workloads need some other value, then we have
>> >> evidence that different workloads need different value.  At that time,
>> >> we can suggest to add an user tunable knob.
>> >>
>> >
>> > The problem is that others are unaware we've set it to 0, and I can't
>> > constantly monitor the linux-mm mailing list. Additionally, it's
>> > possible that you can't always keep an eye on it either.
>>
>> IIUC, they will use the default value.  Then, if there is any
>> performance regression, they can report it.
>
> Now we report it. What is your replyment? "Keep it in your downstream
> kernel." Wow, PANIC again.

This is not all of my reply.  I suggested you to change the default
value too.

>
>>
>> > I believe we should hear Andrew's suggestion. Andrew, what is your opinion?
>>

--
Best Regards,
Huang, Ying





[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux OMAP]     [Linux MIPS]     [eCos]     [Asterisk Internet PBX]     [Linux API]

  Powered by Linux