On 2019/6/24 16:10, Michal Hocko wrote: > On Mon 24-06-19 12:43:26, zhong jiang wrote: >> Recently, I hit an frequent oom issue in linux-4.4 stable with less than 4M free memory after >> the machine boots up. > Is this is a regression? Could you share the oom report? Yep, At least at the small memory machine. I has tested that revert the migrate_highatomic works well. The oom report message is as follows. [ 652.272622] sh invoked oom-killer: gfp_mask=0x26080c0, order=3, oom_score_adj=0 [ 652.272683] CPU: 0 PID: 1748 Comm: sh Tainted: P O 4.4.171 #8 [ 652.345605] Hardware name: Qualcomm (Flattened Device Tree) [ 652.428968] [<c02149d0>] (unwind_backtrace) from [<c02125a4>] (show_stack+0x10/0x14) [ 652.494604] [<c02125a4>] (show_stack) from [<c037fb08>] (dump_stack+0xa0/0xd8) [ 652.590432] [<c037fb08>] (dump_stack) from [<c02bdf58>] (dump_header.constprop.6+0x40/0x15c) [ 652.674793] [<c02bdf58>] (dump_header.constprop.6) from [<c0287504>] (oom_kill_process+0xc4/0x434) [ 652.777934] [<c0287504>] (oom_kill_process) from [<c0287b5c>] (out_of_memory+0x284/0x318) [ 652.883120] [<c0287b5c>] (out_of_memory) from [<c028b990>] (__alloc_pages_nodemask+0x90c/0x9b4) [ 652.982080] [<c028b990>] (__alloc_pages_nodemask) from [<c021aa94>] (copy_process.part.2+0xe4/0x12f0) [ 653.084160] [<c021aa94>] (copy_process.part.2) from [<c021bdf8>] (_do_fork+0xb8/0x2d4) [ 653.196678] [<c021bdf8>] (_do_fork) from [<c021c0d0>] (SyS_clone+0x1c/0x24) [ 653.290424] [<c021c0d0>] (SyS_clone) from [<c020f480>] (ret_fast_syscall+0x0/0x4c) [ 653.452827] Mem-Info: [ 653.466390] active_anon:20377 inactive_anon:187 isolated_anon:0 [ 653.466390] active_file:5087 inactive_file:4825 isolated_file:0 [ 653.466390] unevictable:12 dirty:0 writeback:32 unstable:0 [ 653.466390] slab_reclaimable:636 slab_unreclaimable:1754 [ 653.466390] mapped:5338 shmem:194 pagetables:231 bounce:0 [ 653.466390] free:1086 free_pcp:85 free_cma:0 [ 653.625286] Normal free:4248kB min:1696kB low:2120kB high:2544kB active_anon:81508kB inactive_anon:748kB active_file:20348kB inactive_file:19300kB unevictable:48kB isolated(anon):0kB isolated(file):0kB present:252928kB managed:180496kB mlocked:0kB dirty:0kB writeback:128kB mapped:21352kB shmem:776kB slab_reclaimable:2544kB slab_unreclaimable:7016kB kernel_stack:9856kB pagetables:924kB unstable:0kB bounce:0kB free_pcp:392kB local_pcp:392kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no [ 654.177121] lowmem_reserve[]: 0 0 0 [ 654.462015] Normal: 752*4kB (UME) 128*8kB (UM) 21*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4368kB [ 654.601093] 10132 total pagecache pages [ 654.606655] 63232 pages RAM [ 654.656658] 0 pages HighMem/MovableOnly [ 654.686821] 18108 pages reserved [ 654.731549] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name [ 654.775019] [ 108] 0 108 814 11 6 0 0 0 rcS [ 654.877730] [ 113] 0 113 814 125 6 0 0 0 sh [ 654.978510] [ 116] 0 116 485 16 5 0 0 0 fprefetch [ 655.083679] [ 272] 0 272 1579 83 7 0 0 -1000 sshd [ 655.187516] [ 276] 0 276 7591 518 10 0 0 0 redis-server [ 655.287893] [ 282] 0 282 3495 364 8 0 0 0 callhome [ 655.406249] [ 284] 0 284 1166 322 6 0 0 0 remote_plugin [ 655.524594] [ 292] 0 292 523 113 6 0 0 -17 monitor [ 655.616477] [ 293] 0 293 17958 609 39 0 0 0 cap32 [ 655.724315] [ 296] 0 296 2757 1106 10 0 0 0 confd [ 655.823061] [ 297] 0 297 60183 20757 112 0 0 0 vos.o [ 655.952344] [ 1748] 0 1748 814 92 6 0 0 0 sh ......[ 656.241027] *****************Start oom extend info.***************** >> As the process is created, kernel stack will use the higher order to allocate continuous memory. >> Due to the fragmentabtion, we fails to allocate the memory. And the low memory will result >> in hardly memory compction. hence, it will easily to reproduce the oom. > How get your get such a large fragmentation that you cannot allocate > order-1 pages and compaction is not making any progress? >From the above oom report, we can see that there is not order-2 pages. It wil hardly to allocate kernel stack when creating the process. And we can easily to reproduce the situation when runing some userspace program. But it rarely trigger the oom when It do not introducing the highatomic. we test that in the kernel 3.10. >> But if we use migrate_reserve to reserve at least a pageblock at the boot stage. we can use >> the reserve memory to allocate continuous memory for process when the system is under >> severerly fragmentation. > Well, any reservation is a finite resource so I am not sure how that can > help universally. But your description is quite vague. Could you be more > specific about that workload? Also do you see the same with the current > upstream kernel as well? I just compare the kernel 3.10 with kernel 4.4 in same situation. I do not test the issue in the upstream kernel. and check the /proc/pagetypeinfo after the system boots up. migrate_highatomic is always zero., while migrate_reserve has an pageblock that it can use for high order allocation. Even though go back the migrate_reserve can not solve the issue. but In fact , it did relieve the oom situation. Thanks, zhong jiang
