On Thu 13-12-18 16:37:35, Pingfan Liu wrote: [...] > [ 0.409667] NUMA: Node 1 [mem 0x00000000-0x0009ffff] + [mem 0x00100000-0x7fffffff] -> [mem 0x00000000-0x7fffffff] > [ 0.419885] NUMA: Node 1 [mem 0x00000000-0x7fffffff] + [mem 0x100000000-0x47fffffff] -> [mem 0x00000000-0x47fffffff] > [ 0.430386] NODE_DATA(0) allocated [mem 0x87efd4000-0x87effefff] > [ 0.436352] NODE_DATA(0) on node 5 > [ 0.440124] Initmem setup node 0 [mem 0x0000000000000000-0x0000000000000000] > [ 0.447104] NODE_DATA(1) allocated [mem 0x47ffd5000-0x47fffffff] > [ 0.453110] NODE_DATA(2) allocated [mem 0x87efa9000-0x87efd3fff] > [ 0.459060] NODE_DATA(2) on node 5 > [ 0.462855] Initmem setup node 2 [mem 0x0000000000000000-0x0000000000000000] > [ 0.469809] NODE_DATA(3) allocated [mem 0x87ef7e000-0x87efa8fff] > [ 0.475788] NODE_DATA(3) on node 5 > [ 0.479554] Initmem setup node 3 [mem 0x0000000000000000-0x0000000000000000] > [ 0.486536] NODE_DATA(4) allocated [mem 0x87ef53000-0x87ef7dfff] > [ 0.492518] NODE_DATA(4) on node 5 > [ 0.496280] Initmem setup node 4 [mem 0x0000000000000000-0x0000000000000000] > [ 0.503266] NODE_DATA(5) allocated [mem 0x87ef28000-0x87ef52fff] > [ 0.509281] NODE_DATA(6) allocated [mem 0x87eefd000-0x87ef27fff] > [ 0.515224] NODE_DATA(6) on node 5 > [ 0.518987] Initmem setup node 6 [mem 0x0000000000000000-0x0000000000000000] > [ 0.525974] NODE_DATA(7) allocated [mem 0x87eed2000-0x87eefcfff] > [ 0.531953] NODE_DATA(7) on node 5 > [ 0.535716] Initmem setup node 7 [mem 0x0000000000000000-0x0000000000000000] OK, so we have allocated node_data for all NUMA nodes. Good! > [ 0.542839] Reserving 500MB of memory at 384MB for crashkernel (System RAM: 32314MB) > [ 0.550465] Zone ranges: > [ 0.552927] DMA [mem 0x0000000000001000-0x0000000000ffffff] > [ 0.559081] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] > [ 0.565235] Normal [mem 0x0000000100000000-0x000000087effffff] > [ 0.571388] Device empty > [ 0.574249] Movable zone start for each node > [ 0.578498] Early memory node ranges > [ 0.582049] node 1: [mem 0x0000000000001000-0x000000000008efff] > [ 0.588291] node 1: [mem 0x0000000000090000-0x000000000009ffff] > [ 0.594530] node 1: [mem 0x0000000000100000-0x000000005c3d6fff] > [ 0.600772] node 1: [mem 0x00000000643df000-0x0000000068ff7fff] > [ 0.607011] node 1: [mem 0x000000006c528000-0x000000006fffffff] > [ 0.613251] node 1: [mem 0x0000000100000000-0x000000047fffffff] > [ 0.619493] node 5: [mem 0x0000000480000000-0x000000087effffff] > [ 0.626479] Zeroed struct page in unavailable ranges: 46490 pages > [ 0.626480] Initmem setup node 1 [mem 0x0000000000001000-0x000000047fffffff] > [ 0.655261] Initmem setup node 5 [mem 0x0000000480000000-0x000000087effffff] [...] > [ 1.066324] Built 2 zonelists, mobility grouping off. Total pages: 0 There are 2 zonelists built, but for some reason vm_total_pages is 0 and that is clearly wrong. Because the allocation failure (which later leads to NULL ptr) tells there is quite a lot of memory. One reason might be that the zonelist for memory less nodes is initialized incorrectly. nr_free_zone_pages relies on the local Node zonelist so if the code happened to run on a cpu associated with Node2 then we could indeed got vm_total_pages=0. > [ 1.439440] Node 1 DMA: 2*4kB (U) 2*8kB (U) 2*16kB (U) 3*32kB (U) 2*64kB (U) 2*128kB (U) 2*256kB (U) 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (M) = 15896kB > [ 1.453482] Node 1 DMA32: 2*4kB (M) 1*8kB (M) 1*16kB (M) 2*32kB (M) 3*64kB (M) 2*128kB (M) 3*256kB (M) 3*512kB (M) 2*1024kB (M) 3*2048kB (M) 255*4096kB (M) = 1055520kB > [ 1.468388] Node 1 Normal: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 1*1024kB (U) 1*2048kB (U) 31*4096kB (M) = 131068kB > [ 1.483211] Node 5 Normal: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (U) 1*64kB (U) 1*128kB (U) 1*256kB (U) 1*512kB (U) 1*1024kB (U) 1*2048kB (U) 31*4096kB (M) = 131068kB I am investigating what the hell is going on here. Maybe the former hack to re-initialize memory-less nodes is working around some ordering issues. -- Michal Hocko SUSE Labs
