On Fri, Jun 11, 2021 at 09:54:42AM +0800, Feng Tang wrote:
> 0day robot reported a 9.2% regression for will-it-scale mmap1 test
> case[1], caused by commit 57efa1fe5957 ("mm/gup: prevent gup_fast
> from racing with COW during fork").
>
> Further debug shows the regression is due to that commit changes
> the offset of hot fields 'mmap_lock' inside structure 'mm_struct',
> thus some cache alignment changes.
>
> From the perf data, the contention for 'mmap_lock' is very severe
> and takes around 95% cpu cycles, and it is a rw_semaphore
>
> struct rw_semaphore {
> atomic_long_t count; /* 8 bytes */
> atomic_long_t owner; /* 8 bytes */
> struct optimistic_spin_queue osq; /* spinner MCS lock */
> ...
>
> Before commit 57efa1fe5957 adds the 'write_protect_seq', it
> happens to have a very optimal cache alignment layout, as
> Linus explained:
>
> "and before the addition of the 'write_protect_seq' field, the
> mmap_sem was at offset 120 in 'struct mm_struct'.
>
> Which meant that count and owner were in two different cachelines,
> and then when you have contention and spend time in
> rwsem_down_write_slowpath(), this is probably *exactly* the kind
> of layout you want.
>
> Because first the rwsem_write_trylock() will do a cmpxchg on the
> first cacheline (for the optimistic fast-path), and then in the
> case of contention, rwsem_down_write_slowpath() will just access
> the second cacheline.
>
> Which is probably just optimal for a load that spends a lot of
> time contended - new waiters touch that first cacheline, and then
> they queue themselves up on the second cacheline."
>
> After the commit, the rw_semaphore is at offset 128, which means
> the 'count' and 'owner' fields are now in the same cacheline,
> and causes more cache bouncing.
>
> Currently there are 3 "#ifdef CONFIG_XXX" before 'mmap_lock' which
> will affect its offset:
>
> CONFIG_MMU
> CONFIG_MEMBARRIER
> CONFIG_HAVE_ARCH_COMPAT_MMAP_BASES
>
> The layout above is on 64 bits system with 0day's default kernel
> config (similar to RHEL-8.3's config), in which all these 3 options
> are 'y'. And the layout can vary with different kernel configs.
>
> Relayouting a structure is usually a double-edged sword, as sometimes
> it can helps one case, but hurt other cases. For this case, one
> solution is, as the newly added 'write_protect_seq' is a 4 bytes long
> seqcount_t (when CONFIG_DEBUG_LOCK_ALLOC=n), placing it into an
> existing 4 bytes hole in 'mm_struct' will not change other fields'
> alignment, while restoring the regression.
>
> [1]. https://lore.kernel.org/lkml/20210525031636.GB7744@xsang-OptiPlex-9020/
> Reported-by: kernel test robot <oliver.sang@xxxxxxxxx>
> Signed-off-by: Feng Tang <feng.tang@xxxxxxxxx>
> Reviewed-by: John Hubbard <jhubbard@xxxxxxxxxx>
> Cc: Jason Gunthorpe <jgg@xxxxxxxxxx>
> Cc: Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx>
> Cc: Peter Xu <peterx@xxxxxxxxxx>
> ---
> include/linux/mm_types.h | 27 ++++++++++++++++++++-------
> 1 file changed, 20 insertions(+), 7 deletions(-)
It seems Ok to me, but didn't we earlier add the has_pinned which
would have changed the layout too? Are we chasing performance delta's
nobody cares about?
Still it is mechanically fine, so:
Reviewed-by: Jason Gunthorpe <jgg@xxxxxxxxxx>
Jason
