----- Original Message -----
> On 13/12/2018 10:26, Jan Stancek wrote:
> >
> >
> > ----- Original Message -----
> >> + cc'ing linuxarm@xxxxxxxxxx
> >>
> >> It seems that we're spending much time in cache invalidate.
> >>
> >> When you say 4 nodes, does that mean memory on all 4 nodes?
> >
> > Correct:
> >
> > # numactl -H
> > available: 4 nodes (0-3)
> > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
> > node 0 size: 65304 MB
> > node 0 free: 59939 MB
> > node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
> > node 1 size: 65404 MB
> > node 1 free: 64419 MB
> > node 2 cpus: 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
> > node 2 size: 65404 MB
> > node 2 free: 64832 MB
> > node 3 cpus: 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63
> > node 3 size: 65403 MB
> > node 3 free: 64805 MB
> > node distances:
> > node 0 1 2 3
> > 0: 10 15 20 20
> > 1: 15 10 20 20
> > 2: 20 20 10 15
> > 3: 20 20 15 10
> >
>
> Thanks. I assume that you're using 64K pages, but I would not say that
> would make a difference.
Yes, pages are 64k.
>
> Have you tested other arm64 systems and found similar?
I tried couple just now, but I don't see slowdown on other arm64 systems:
1. Cavium ThunderX (CN8890-1800BG2601-AAP-PR-Y-G)
...
[pid 8756] 10:25:14 migrate_pages(0, 8, [0x0000000000000002], [0x0000000000000001]) = 0 <0.000261>
[pid 8756] 10:25:14 migrate_pages(0, 8, [0x0000000000000001], [0x0000000000000002]) = 13 <0.015828>
2. GIGABYTE MT70-HD0 (CN8890-2000BG2601-AAP-PR-Y-G)
...
[pid 8829] 10:30:05 migrate_pages(0, 8, [0x0000000000000002], [0x0000000000000001]) = 0 <0.002318>
[pid 8829] 10:30:05 migrate_pages(0, 8, [0x0000000000000001], [0x0000000000000002]) = 0 <0.002277>
---
I'll query our lab guys about background info for that huawei-t2280.
For now, here's some info from cpuinfo/lshw:
processor : 0
BogoMIPS : 100.00
Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer : 0x41
CPU architecture: 8
CPU variant : 0x0
CPU part : 0xd08
CPU revision : 2
*-cpu:0
description: CPU
product: ARM (To be filled by O.E.M.)
vendor: Hisilicon
physical id: 1d
bus info: cpu@0
version: Hi1616
serial: To be filled by O.E.M.
slot: CPU01
size: 2400MHz
capacity: 2400MHz
clock: 50MHz
capabilities: lm
configuration: cores=32 enabledcores=32 threads=32
Regards,
Jan
>
> We will check this test ourselves.
>
> Cheers,
> John
>
> >>
> >> Thanks,
> >> John
> >>
> >> On 12/12/2018 15:41, Jan Stancek wrote:
> >>> Hi,
> >>>
> >>> I'm observing migrate_pages() taking quite long time on arm64
> >>> system (Huawei TaiShan 2280, 4 nodes, 64 CPUs). I'm using 4.20.0-rc6,
> >>> but it's reproducible with older kernels (4.14) as well.
> >>>
> >>> The test (see [1] below), is a trivial C application, that migrates
> >>> current process from one node to another. More complicated example
> >>> is also LTP's migrate_pages03, where this has been originally reported.
> >>>
> >>> It takes 2+ seconds to migrate process from one node to another:
> >>> # strace -f -t -T ./a.out
> >>> ...
> >>> [pid 13754] 10:17:13 migrate_pages(0, 8, [0x0000000000000002],
> >>> [0x0000000000000001]) = 1 <0.058115>
> >>> [pid 13754] 10:17:13 migrate_pages(0, 8, [0x0000000000000001],
> >>> [0x0000000000000002]) = 12 <2.348186>
> >>> [pid 13754] 10:17:16 migrate_pages(0, 8, [0x0000000000000002],
> >>> [0x0000000000000001]) = 1 <0.057889>
> >>> [pid 13754] 10:17:16 migrate_pages(0, 8, [0x0000000000000001],
> >>> [0x0000000000000002]) = 10 <2.194890>
> >>> ...
> >>>
> >>> This scales with number of children. For example with MAXCHILD 1000,
> >>> it takes ~33 seconds:
> >>> # strace -f -t -T ./a.out
> >>> ...
> >>> [pid 13773] 10:17:55 migrate_pages(0, 8, [0x0000000000000001],
> >>> [0x0000000000000002]) = 11 <33.615550>
> >>> [pid 13773] 10:18:29 migrate_pages(0, 8, [0x0000000000000002],
> >>> [0x0000000000000001]) = 2 <5.460270>
> >>> ...
> >>>
> >>> It appears to be related to migration of shared pages, presumably
> >>> executable code of glibc.
> >>>
> >>> If I run [1] without CAP_SYS_NICE, it completes very quickly:
> >>> # sudo -u nobody strace -f -t -T ./a.out
> >>> ...
> >>> [pid 14847] 10:24:57 migrate_pages(0, 8, [0x0000000000000001],
> >>> [0x0000000000000002]) = 0 <0.000172>
> >>> [pid 14847] 10:24:57 migrate_pages(0, 8, [0x0000000000000002],
> >>> [0x0000000000000001]) = 0 <0.000091>
> >>> [pid 14847] 10:24:57 migrate_pages(0, 8, [0x0000000000000001],
> >>> [0x0000000000000002]) = 0 <0.000074>
> >>> [pid 14847] 10:24:57 migrate_pages(0, 8, [0x0000000000000002],
> >>> [0x0000000000000001]) = 0 <0.000069>
> >>> ...
> >>>
> >>>
> >>> Looking at perf, most of time is spent invalidating icache.
> >>>
> >>> - 100.00% 0.00% a.out [kernel.kallsyms] [k] __sys_trace_return
> >>> - __sys_trace_return
> >>> - 100.00% __se_sys_migrate_pages
> >>> do_migrate_pages.part.9
> >>> - migrate_pages
> >>> - 99.92% rmap_walk
> >>> - 99.92% rmap_walk_file
> >>> - 99.90% remove_migration_pte
> >>> - 99.85% __sync_icache_dcache
> >>> __flush_cache_user_range
> >>>
> >>> Percent│ nop
> >>> │ ubfx x3, x3, #16, #4
> >>> │ mov x2, #0x4 // #4
> >>> │ lsl x2, x2, x3
> >>> │ sub x3, x2, #0x1
> >>> │ bic x4, x0, x3
> >>> 1.82 │ dc cvau, x4
> >>> │ add x4, x4, x2
> >>> │ cmp x4, x1
> >>> │ → b.cc 0xffff00000809efc8 // b.lo, b.ul, fffff7f61067
> >>> │ dsb ish
> >>> │ nop
> >>> 0.07 │ nop
> >>> │ mrs x3, ctr_el0
> >>> │ nop
> >>> │ and x3, x3, #0xf
> >>> │ mov x2, #0x4 // #4
> >>> │ lsl x2, x2, x3
> >>> │ sub x3, x2, #0x1
> >>> │ bic x3, x0, x3
> >>> 96.17 │ ic ivau, x3
> >>> │ add x3, x3, x2
> >>> │ cmp x3, x1
> >>> │ → b.cc 0xffff00000809f000 // b.lo, b.ul, fffff7f61067
> >>> 0.10 │ dsb ish
> >>> │ isb
> >>> 1.85 │ mov x0, #0x0 // #0
> >>> │78: ← ret
> >>> │ mov x0, #0xfffffffffffffff2 // #-14
> >>> │ ↑ b 78
> >>>
> >>> Regards,
> >>> Jan
> >>>
> >>> [1]
> >>> ----- 8< -----
> >>> #include <signal.h>
> >>> #include <stdio.h>
> >>> #include <stdlib.h>
> >>> #include <unistd.h>
> >>> #include <sys/syscall.h>
> >>>
> >>> #define MAXCHILD 10
> >>>
> >>> int main(void)
> >>> {
> >>> long node1 = 1, node2 = 2;
> >>> int i, child;
> >>> int pids[MAXCHILD];
> >>>
> >>> for (i = 0; i < MAXCHILD; i++) {
> >>> child = fork();
> >>> if (child == 0) {
> >>> sleep(600);
> >>> exit(0);
> >>> }
> >>> pids[i] = child;
> >>> }
> >>>
> >>> for (i = 0; i < 5; i++) {
> >>> syscall(__NR_migrate_pages, 0, 8, &node1, &node2);
> >>> syscall(__NR_migrate_pages, 0, 8, &node2, &node1);
> >>> }
> >>>
> >>> for (i = 0; i < MAXCHILD; i++) {
> >>> kill(pids[i], SIGKILL);
> >>> }
> >>>
> >>> return 0;
> >>> }
> >>> ----- >8 -----
> >>>
> >>> _______________________________________________
> >>> linux-arm-kernel mailing list
> >>> linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
> >>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> >>>
> >>
> >>
> >>
> >
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@xxxxxxxxxxxxxxxxxxx
> > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> >
>
>
>
