I'm seeing this on an ia64 platform--HP rx8640--running the numactl
package regression test. On ia64 a "NaT Consumption" [NaT = "not a
thing"] usually means a bogus pointer. I verified that it also occurs
on 2.6.30-rc3-mmotm-090424-1814. The regression test runs to completion
on a 4-node x86_64 platform for both the 04/27 and 04/28 mmotm kernels.
The bug occurs right after the test suite issues the message:
"testing numactl --interleave=all memhog 15728640"
-------------------------------
Console log:
numactl[7821]: NaT consumption 2216203124768 [2]
Modules linked in: ipv6 nfs lockd fscache nfs_acl auth_rpcgss sunrpc vfat fat dm_mirror dm_multipath scsi_dh pci_slot parport_pc lp parport sg sr_mod cdrom button e1000 tg3 libphy dm_region_hash dm_log dm_mod sym53c8xx mptspi mptscsih mptbase scsi_transport_spi sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd [last unloaded: freq_table]
Pid: 7821, CPU 25, comm: numactl
psr : 0000121008022038 ifs : 8000000000000004 ip : [<a00000010014ec91>] Not tainted (2.6.30-rc3-mmotm-090428-1631)
ip is at next_zones_zonelist+0x31/0x120
unat: 0000000000000000 pfs : 000000000000038b rsc : 0000000000000003
rnat: 0000001008022038 bsps: e0000702c094e000 pr : 0000000000659a65
ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c0270033f
csd : 0000000000000000 ssd : 0000000000000000
b0 : a00000010018dd30 b6 : a0000001000414c0 b7 : a000000100474660
f6 : 1003e3d64d824615cef9c f7 : 1003e9e3779b97f4a7c16
f8 : 1003e0a00000010071e2a f9 : 1003e0000000000000001
f10 : 1003e0000000000000003 f11 : 1003e8208208208208209
r1 : a000000100c647c0 r2 : 0000000000001f90 r3 : 0000000000000000
r8 : 0000000000001f88 r9 : 0000000000000000 r10 : 0000000000000000
r11 : 0000000000000000 r12 : e0000803859eaad0 r13 : e0000803859e8000
r14 : 0000000000000000 r15 : ffffffffffffff80 r16 : e000078000113c38
r17 : 0000000000000000 r18 : 0000000000000006 r19 : e000078000113c08
r20 : 0000000000000000 r21 : e000078000113c00 r22 : 0000000000000000
r23 : ffffffffffff04d8 r24 : e0000803859e8cb8 r25 : 000000000000033c
r26 : e0000803859eab60 r27 : e0000803859eab54 r28 : e0000803859eab58
r29 : 000000007fffffff r30 : 000000000000033c r31 : 000000000000033c
Call Trace:
[<a000000100015660>] show_stack+0x40/0xa0
sp=e0000803859ec690 bsp=e0000803859ec670
[<a000000100015f90>] show_regs+0x870/0x8c0
sp=e0000803859ec860 bsp=e0000803859ec618
[<a0000001000398b0>] die+0x1b0/0x2c0
sp=e0000803859ec860 bsp=e0000803859ec5c8
[<a000000100039a10>] die_if_kernel+0x50/0x80
sp=e0000803859ec860 bsp=e0000803859ec598
This is all I see. Apparently system locked up.
------------------
Here's the memory init debug info:
Zone PFN ranges:
DMA 0x00000001 -> 0x00040000
Normal 0x00040000 -> 0x220ff000
Movable zone start PFN for each node
early_node_map[6] active PFN ranges
4: 0x00000001 -> 0x00007ffa
0: 0x1c008000 -> 0x1c0fec00
1: 0x1e000000 -> 0x1e0ff000
2: 0x20000000 -> 0x200ff000
3: 0x22000000 -> 0x220fef67
3: 0x220fefa0 -> 0x220feff6
mminit::pageflags_layout_widths Section 0 Node 6 Zone 2 Flags 23
mminit::pageflags_layout_shifts Section 20 Node 6 Zone 2
mminit::pageflags_layout_offsets Section 0 Node 58 Zone 56
mminit::pageflags_layout_zoneid Zone ID: 56 -> 64
mminit::pageflags_layout_usage location: 64 -> 56 unused 56 -> 23 flags 23 -> 0
On node 0 totalpages: 1010688
Normal zone: 3948 pages used for memmap
Normal zone: 1006740 pages, LIFO batch:7
mminit::memmap_init Initialising map node 0 zone 1 pfns 469794816 -> 470805504
On node 1 totalpages: 1044480
Normal zone: 4080 pages used for memmap
Normal zone: 1040400 pages, LIFO batch:7
mminit::memmap_init Initialising map node 1 zone 1 pfns 503316480 -> 504360960
On node 2 totalpages: 1044480
Normal zone: 4080 pages used for memmap
Normal zone: 1040400 pages, LIFO batch:7
mminit::memmap_init Initialising map node 2 zone 1 pfns 536870912 -> 537915392
On node 3 totalpages: 1044413
Normal zone: 4080 pages used for memmap
Normal zone: 1040333 pages, LIFO batch:7
mminit::memmap_init Initialising map node 3 zone 1 pfns 570425344 -> 571469814
On node 4 totalpages: 32761
DMA zone: 128 pages used for memmap
DMA zone: 0 pages reserved
DMA zone: 32633 pages, LIFO batch:7
mminit::memmap_init Initialising map node 4 zone 0 pfns 1 -> 32762
mminit::zonelist general 0:Normal = 0:Normal 1:Normal 2:Normal 3:Normal 4:DMA
mminit::zonelist thisnode 0:Normal = 0:Normal
mminit::zonelist general 1:Normal = 1:Normal 2:Normal 3:Normal 0:Normal 4:DMA
mminit::zonelist thisnode 1:Normal = 1:Normal
mminit::zonelist general 2:Normal = 2:Normal 3:Normal 0:Normal 1:Normal 4:DMA
mminit::zonelist thisnode 2:Normal = 2:Normal
mminit::zonelist general 3:Normal = 3:Normal 0:Normal 1:Normal 2:Normal 4:DMA
mminit::zonelist thisnode 3:Normal = 3:Normal
mminit::zonelist general 4:DMA = 4:DMA
mminit::zonelist thisnode 4:DMA = 4:DMA
Built 5 zonelists in Zone order, mobility grouping on. Total pages: 4160506
Note that this platform has a small [~512MB] pseudo-node #4 that
contains DMA only. Here's the 'numactl --hardware' output:
available: 5 nodes (0-4)
node 0 size: 15792 MB
node 0 free: 14908 MB
node 1 size: 16320 MB
node 1 free: 15985 MB
node 2 size: 16320 MB
node 2 free: 16106 MB
node 3 size: 16318 MB
node 3 free: 16146 MB
node 4 size: 511 MB
node 4 free: 495 MB
node distances:
node 0 1 2 3 4
0: 10 17 17 17 14
1: 17 10 17 17 14
2: 17 17 10 17 14
3: 17 17 17 10 14
4: 14 14 14 14 10
If I create a cpuset with "mems" 0-3 -- i.e., eliminate the dma-only
node 4 -- I do not hit the this "Nat Consumption" bug. The x86_64 test
platform doesn't have this "feature".
I suspect that the page alloc optimizations are making assumptions that
aren't true for this platform. I know we had to muck around quite a
bit to get this all to work in the "memoryless nodes" and "two zonelist"
patches a while back.
I'll try to bisect to specific patch--probably tomorrow.
Regards,
Lee
--
To unsubscribe from this list: send the line "unsubscribe linux-numa" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
