System is an intel dual socket Xeon E5620, 7500/5520/5500/X58 ICH10
family according to lspci. Anyways 4.8.4 OOM'd while I was gone. I'll
download the current 4.9rc and reboot, but in the mean time here's
xxd, vmstat & kern.log output:
8532039 0000000000000000
9324 0100000000000000
2226 0200000000000000
405 0300000000000000
80 0400000000000000
34 0500000000000000
48 0600000000000000
17 0700000000000000
17 0800000000000000
32 0900000000000000
19 0a00000000000000
1 0c00000000000000
1 0d00000000000000
1 0e00000000000000
12 1000000000000000
8 1100000000000000
32 1200000000000000
10 1300000000000000
2 1400000000000000
11 1500000000000000
12 1600000000000000
7 1700000000000000
3 1800000000000000
5 1900000000000000
6 1a00000000000000
11 1b00000000000000
22 1c00000000000000
3 1d00000000000000
19 1e00000000000000
21 1f00000000000000
18 2000000000000000
28 2100000000000000
40 2200000000000000
38 2300000000000000
85 2400000000000000
59 2500000000000000
40520 81ffffffffffffff
/proc/vmstat:
nr_free_pages 60965
nr_zone_inactive_anon 4646
nr_zone_active_anon 3265
nr_zone_inactive_file 633882
nr_zone_active_file 7017458
nr_zone_unevictable 0
nr_zone_write_pending 0
nr_mlock 0
nr_slab_reclaimable 299205
nr_slab_unreclaimable 195497
nr_page_table_pages 935
nr_kernel_stack 4976
nr_bounce 0
numa_hit 3577063288
numa_miss 541393191
numa_foreign 541393191
numa_interleave 19415
numa_local 3577063288
numa_other 0
nr_free_cma 0
nr_inactive_anon 4646
nr_active_anon 3265
nr_inactive_file 633882
nr_active_file 7017458
nr_unevictable 0
nr_isolated_anon 0
nr_isolated_file 0
nr_pages_scanned 0
workingset_refault 42685891
workingset_activate 15247281
workingset_nodereclaim 26375216
nr_anon_pages 5067
nr_mapped 5630
nr_file_pages 7654746
nr_dirty 0
nr_writeback 0
nr_writeback_temp 0
nr_shmem 2504
nr_shmem_hugepages 0
nr_shmem_pmdmapped 0
nr_anon_transparent_hugepages 0
nr_unstable 0
nr_vmscan_write 5243750485
nr_vmscan_immediate_reclaim 4207633857
nr_dirtied 1839143430
nr_written 1832626107
nr_dirty_threshold 1147728
nr_dirty_background_threshold 151410
pgpgin 166731189
pgpgout 7328142335
pswpin 98608
pswpout 117794
pgalloc_dma 29504
pgalloc_dma32 1006726216
pgalloc_normal 5275218188
pgalloc_movable 0
allocstall_dma 0
allocstall_dma32 0
allocstall_normal 36461
allocstall_movable 5867
pgskip_dma 0
pgskip_dma32 0
pgskip_normal 6417890
pgskip_movable 0
pgfree 6309223401
pgactivate 35076483
pgdeactivate 63556974
pgfault 35753842
pgmajfault 69126
pglazyfreed 0
pgrefill 70008598
pgsteal_kswapd 3567289713
pgsteal_direct 5878057
pgscan_kswapd 9059309872
pgscan_direct 4239367903
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 102916
slabs_scanned 460790262
kswapd_inodesteal 9130243
kswapd_low_wmark_hit_quickly 10634373
kswapd_high_wmark_hit_quickly 7348173
pageoutrun 18349115
pgrotated 16291322
drop_pagecache 0
drop_slab 0
pgmigrate_success 18912908
pgmigrate_fail 63382146
compact_migrate_scanned 2986269789
compact_free_scanned 190451505123
compact_isolated 109549437
compact_stall 3544
compact_fail 8
compact_success 3536
compact_daemon_wake 1403515
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 12473
unevictable_pgs_scanned 0
unevictable_pgs_rescued 11979
unevictable_pgs_mlocked 14556
unevictable_pgs_munlocked 14556
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
thp_fault_alloc 0
thp_fault_fallback 0
thp_collapse_alloc 0
thp_collapse_alloc_failed 0
thp_file_alloc 0
thp_file_mapped 0
thp_split_page 0
thp_split_page_failed 0
thp_deferred_split_page 0
thp_split_pmd 0
thp_zero_page_alloc 0
thp_zero_page_alloc_failed 0
kern.log OOM message:
[737778.724194] snmpd invoked oom-killer:
gfp_mask=0x24200ca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[737778.724246] snmpd cpuset=/ mems_allowed=0-1
[737778.724278] CPU: 15 PID: 2976 Comm: snmpd Tainted: G W I 4.8.4 #1
[737778.724352] 0000000000000000 ffffffff81292069 ffff88041e043c48
ffff88041e043c48
[737778.724403] ffffffff8118d1f6 ffff88041dd70fc0 ffff88041e043c48
000000000136236f
[737778.724454] ffffffff8170e11e 0000000000000001 ffffffff8112a700
000000000000030f
[737778.724505] Call Trace:
[737778.724533] [<ffffffff81292069>] ? dump_stack+0x46/0x5d
[737778.727077] [<ffffffff8118d1f6>] ? dump_header.isra.16+0x56/0x185
[737778.727108] [<ffffffff8112a700>] ? oom_kill_process+0x210/0x3c0
[737778.727136] [<ffffffff8112ac4b>] ? out_of_memory+0x34b/0x420
[737778.727165] [<ffffffff8112fcca>] ? __alloc_pages_nodemask+0xd9a/0xde0
[737778.727195] [<ffffffff811768e1>] ? alloc_pages_vma+0xc1/0x240
[737778.727223] [<ffffffff81126512>] ? pagecache_get_page+0x22/0x230
[737778.727253] [<ffffffff81169f44>] ? __read_swap_cache_async+0x104/0x180
[737778.727282] [<ffffffff81169fcf>] ? read_swap_cache_async+0xf/0x30
[737778.727311] [<ffffffff8116a0dc>] ? swapin_readahead+0xec/0x1a0
[737778.727340] [<ffffffff81156270>] ? do_swap_page+0x420/0x5c0
[737778.727369] [<ffffffff813f36d8>] ? SYSC_recvfrom+0xa8/0x110
[737778.727397] [<ffffffff81157a39>] ? handle_mm_fault+0x629/0xe30
[737778.727426] [<ffffffff81048fc5>] ? __do_page_fault+0x1b5/0x480
[737778.727456] [<ffffffff814fbaa2>] ? page_fault+0x22/0x30
[737778.727497] Mem-Info:
[737778.727524] active_anon:24 inactive_anon:49 isolated_anon:0
[737778.727524] active_file:6920154 inactive_file:798043 isolated_file:576
[737778.727524] unevictable:0 dirty:800528 writeback:1307 unstable:0
[737778.727524] slab_reclaimable:264367 slab_unreclaimable:193348
[737778.727524] mapped:4063 shmem:0 pagetables:1719 bounce:0
[737778.727524] free:39225 free_pcp:47 free_cma:0
[737778.727677] Node 0 active_anon:16kB inactive_anon:76kB
active_file:14249324kB inactive_file:1296908kB unevictable:0kB
isolated(anon):0kB isolated(file):1920kB mapped:10432kB
dirty:1308528kB writeback:0kB shmem:0kB shmem_thp: 0kB
shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB
pages_scanned:23557303 all_unreclaimable? yes
[737778.727806] Node 1 active_anon:80kB inactive_anon:120kB
active_file:13431292kB inactive_file:1895264kB unevictable:0kB
isolated(anon):0kB isolated(file):384kB mapped:5820kB dirty:1893584kB
writeback:5228kB shmem:0kB shmem_thp: 0kB shmem_pmdmapped: 0kB
anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:25598673
all_unreclaimable? yes
[737778.727930] Node 0 Normal free:44864kB min:45192kB low:61736kB
high:78280kB active_anon:16kB inactive_anon:76kB
active_file:14249324kB inactive_file:1296908kB unevictable:0kB
writepending:1308528kB present:16777216kB managed:16544856kB
mlocked:0kB slab_reclaimable:562644kB slab_unreclaimable:317504kB
kernel_stack:3840kB pagetables:3672kB bounce:0kB free_pcp:20kB
local_pcp:0kB free_cma:0kB
[737778.728066] lowmem_reserve[]: 0 0 0 0
[737778.728100] Node 1 DMA free:15896kB min:40kB low:52kB high:64kB
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB
unevictable:0kB writepending:0kB present:15996kB managed:15896kB
mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB
kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
free_cma:0kB
[737778.728228] lowmem_reserve[]: 0 3216 16044 16044
[737778.728263] Node 1 DMA32 free:60300kB min:8996kB low:12288kB
high:15580kB active_anon:4kB inactive_anon:4kB active_file:2660988kB
inactive_file:474956kB unevictable:0kB writepending:475116kB
present:3378660kB managed:3304720kB mlocked:0kB
slab_reclaimable:83612kB slab_unreclaimable:16668kB kernel_stack:320kB
pagetables:16kB bounce:0kB free_pcp:4kB local_pcp:4kB free_cma:0kB
[737778.728397] lowmem_reserve[]: 0 0 12827 12827
[737778.728431] Node 1 Normal free:35840kB min:35876kB low:49008kB
high:62140kB active_anon:76kB inactive_anon:116kB
active_file:10770304kB inactive_file:1420308kB unevictable:0kB
writepending:1423696kB present:13369344kB managed:13135424kB
mlocked:0kB slab_reclaimable:411212kB slab_unreclaimable:439220kB
kernel_stack:2864kB pagetables:3188kB bounce:0kB free_pcp:164kB
local_pcp:36kB free_cma:0kB
[737778.728568] lowmem_reserve[]: 0 0 0 0
[737778.728601] Node 0 Normal: 11208*4kB (UME) 4*8kB (U) 0*16kB 0*32kB
0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44864kB
[737778.728686] Node 1 DMA: 0*4kB 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB
(U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB
(M) = 15896kB
[737778.728786] Node 1 DMA32: 11759*4kB (UME) 1658*8kB (UM) 0*16kB
0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB =
60300kB
[737778.728875] Node 1 Normal: 7984*4kB (UME) 470*8kB (UME) 3*16kB (U)
3*32kB (UM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB
= 35840kB
[737778.728973] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[737778.729019] Node 0 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[737778.729065] Node 1 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=1048576kB
[737778.729111] Node 1 hugepages_total=0 hugepages_free=0
hugepages_surp=0 hugepages_size=2048kB
[737778.729156] 7718841 total pagecache pages
[737778.729179] 68 pages in swap cache
[737778.729202] Swap cache stats: add 193888, delete 193820, find 160188/213014
[737778.729231] Free swap = 48045076kB
[737778.729254] Total swap = 48300028kB
[737778.729277] 8385304 pages RAM
[737778.729299] 0 pages HighMem/MovableOnly
[737778.729322] 135080 pages reserved
[737778.729344] 0 pages hwpoisoned
[737778.729365] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds
swapents oom_score_adj name
[737778.729417] [ 1927] 0 1927 9941 447 22 3
299 -1000 systemd-udevd
[737778.729465] [ 2812] 0 2812 9289 412 23 4
161 0 rpcbind
[737778.729512] [ 2836] 102 2836 9320 414 23 3
151 0 rpc.statd
[737778.729560] [ 2851] 104 2851 162257 276 75 3
7489 0 apt-cacher-ng
[737778.729608] [ 2856] 0 2856 13796 345 31 3
167 -1000 sshd
[737778.729655] [ 2857] 0 2857 64668 504 27 4
355 0 rsyslogd
[737778.729702] [ 2858] 0 2858 6876 518 18 3
83 0 cron
[737778.729748] [ 2859] 0 2859 4756 360 14 3
44 0 atd
[737778.729795] [ 2860] 0 2860 7059 523 18 3
591 0 smartd
[737778.729842] [ 2864] 0 2864 7082 559 19 4
96 0 systemd-logind
[737778.729890] [ 2865] 106 2865 10563 528 24 3
110 -900 dbus-daemon
[737778.729938] [ 2925] 0 2925 3604 421 12 3
38 0 agetty
[737778.729985] [ 2974] 0 2974 4852 365 13 3
73 0 irqbalance
[737778.730032] [ 2976] 105 2976 14299 496 29 3
1534 0 snmpd
[737778.730078] [ 2992] 0 2992 3180 227 11 3
38 0 mcelog
[737778.730125] [ 3095] 0 3095 26571 344 43 3
259 0 sfcbd
[737778.730172] [ 3172] 0 3172 20392 261 40 3
236 0 sfcbd
[737778.730219] [ 3248] 0 3248 22441 0 41 3
238 0 sfcbd
[737778.730265] [ 3249] 0 3249 39376 155 44 3
357 0 sfcbd
[737778.730312] [ 3450] 0 3450 39377 104 44 3
244 0 sfcbd
[737778.730359] [ 3467] 0 3467 58324 301 46 3
284 0 sfcbd
[737778.730405] [ 3548] 0 3548 262686 643 66 4
4097 0 dsm_sa_datamgrd
[737778.730453] [ 3563] 101 3563 13312 403 29 3
162 0 exim4
[737778.730499] [ 3576] 107 3576 7293 493 19 3
148 0 ntpd
[737778.730546] [ 3585] 0 3585 61531 577 117 3
496 0 winbindd
[737778.730593] [ 3586] 0 3586 61531 578 118 3
512 0 winbindd
[737778.730640] [ 3651] 0 3651 48584 566 36 3
487 0 dsm_sa_eventmgr
[737778.730688] [ 3674] 0 3674 99593 576 47 3
1402 0 dsm_sa_snmpd
[737778.730736] [ 3717] 0 3717 7923 285 18 3
115 0 dsm_om_connsvcd
[737778.730784] [ 3718] 0 3718 740234 1744 202 7
30685 0 dsm_om_connsvcd
[737778.730832] [ 3736] 0 3736 178651 0 55 3
3789 0 dsm_sa_datamgrd
[737778.730880] [ 4056] 0 4056 26472 498 57 3
252 0 sshd
[737778.730926] [ 4060] 1000 4060 8973 501 23 3
184 0 systemd
[737778.730973] [ 4061] 1000 4061 15702 0 34 4
612 0 (sd-pam)
[737778.731020] [ 4063] 1000 4063 26472 158 54 3
260 0 sshd
[737778.731067] [ 4064] 1000 4064 6041 739 16 3
686 0 bash
[737778.731113] [ 4083] 1000 4083 16853 493 37 3
128 0 su
[737778.731160] [ 4084] 0 4084 5501 756 15 3
160 0 bash
[737778.731207] [15150] 0 15150 3309 678 10 3
57 0 run_mirror.sh
[737778.731256] [24296] 0 24296 1450 139 8 3
23 0 flock
[737778.731302] [24297] 0 24297 9576 622 22 3
3990 0 rsync
[737778.731349] [24298] 0 24298 7552 541 18 3
1073 0 rsync
[737778.731395] [24299] 0 24299 9522 401 22 3
2416 0 rsync
[737778.731445] [25910] 0 25910 10257 522 23 3
81 0 systemd-journal
[737778.731494] [25940] 0 25940 16365 617 37 3
126 0 cron
[737778.731540] Out of memory: Kill process 3718 (dsm_om_connsvcd)
score 1 or sacrifice child
[737778.731644] Killed process 3718 (dsm_om_connsvcd)
total-vm:2960936kB, anon-rss:0kB, file-rss:6976kB, shmem-rss:0kB
[737778.768375] oom_reaper: reaped process 3718 (dsm_om_connsvcd), now
anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
On Fri, Nov 4, 2016 at 5:00 PM, Vlastimil Babka <vbabka@xxxxxxx> wrote:
> On 11/04/2016 03:13 PM, E V wrote:
>> After the system panic'd yesterday I booted back into 4.8.4 and
>> restarted the rsync's. I'm away on vacation next week, so when I get
>> back I'll get rc4 or rc5 and try again. In the mean time here's data
>> from the system running 4.8.4 without problems for about a day. I'm
>> not familiar with xxd and didn't see a -e option, so used -E:
>> xxd -E -g8 -c8 /proc/kpagecount | cut -d" " -f2 | sort | uniq -c
>> 8258633 0000000000000000
>> 216440 0100000000000000
>
> The lack of -e means it's big endian, which is not a big issue. So here
> most of memory is free, some pages have just one pin, and only
> relatively few have more. The vmstats also doesn't show anything bad, so
> we'll have to wait if something appears within the week, or after you
> try 4.9 again. Thanks.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>