Over-eager swapping

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



We run a number of relatively large x86-64 hosts with twenty or so qemu-kvm
virtual machines on each of them, and I'm have some trouble with over-eager
swapping on some of the machines. This is resulting in load spikes during the
swapping and customer reports of very poor response latency from the virtual
machines which have been swapped out, despite the hosts apparently having
large amounts of free memory, and running fine if swap is turned off.


All of the hosts are currently running a 3.1.4 or 3.2.2 kernel and have ksm
enabled with 64GB of RAM and 2x eight-core AMD Opteron 6128 processors.
However, we have seen this same problem since 2010 on a 2.6.32.7 kernel and
older hardware - see http://marc.info/?l=linux-mm&m=128075337008943
(previous helpful contributors cc:ed here - thanks).

We have /proc/sys/vm/swappiness set to 0. The kernel config is here:
http://users.org.uk/config-3.1.4


The rrd graphs at http://imgur.com/a/Fklxr show a typical incident.

We estimate memory used from /proc/meminfo as:

  = MemTotal - MemFree - Buffers + SwapTotal - SwapFree

The first rrd shows memory used increasing as a VM starts, but not getting
near the 64GB of physical RAM.

The second rrd shows the heavy swapping this VM start caused.

The third rrd shows a multi-gigabyte jump in swap used = SwapTotal - SwapFree

The fourth rrd shows the large load spike (from 1 to 15) caused by this swap
storm.


It is obviously hard to capture all of the relevant data actually during an
incident. However, as of this morning, the relevant stats are as below.

Any help much appreciated! Our strong belief is that there is unnecessary
swapping going on here, and causing these load spikes. We would like to run
with swap for real out-of-memory situations, but at present it is causing
these kind of load spikes on machines which run completely happily with swap
disabled.

Thanks,

Richard.


# cat /proc/meminfo
MemTotal:       65915384 kB
MemFree:          271104 kB
Buffers:        36274368 kB
Cached:            31048 kB
SwapCached:      1830860 kB
Active:         30594144 kB
Inactive:       32295972 kB
Active(anon):   21883428 kB
Inactive(anon):  4695308 kB
Active(file):    8710716 kB
Inactive(file): 27600664 kB
Unevictable:        6740 kB
Mlocked:            6740 kB
SwapTotal:      33054708 kB
SwapFree:       30067948 kB
Dirty:              1044 kB
Writeback:             0 kB
AnonPages:      24962708 kB
Mapped:             7320 kB
Shmem:                48 kB
Slab:            2210964 kB
SReclaimable:    1013272 kB
SUnreclaim:      1197692 kB
KernelStack:        6816 kB
PageTables:       129248 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    66012400 kB
Committed_AS:   67375852 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      259380 kB
VmallocChunk:   34308695568 kB
HardwareCorrupted:     0 kB
AnonHugePages:    155648 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:         576 kB
DirectMap2M:     2095104 kB
DirectMap1G:    65011712 kB

# cat /proc/sys/vm/zone_reclaim_mode
0

# cat /proc/sys/vm/min_unmapped_ratio
1

# cat /proc/slabinfo
slabinfo - version: 2.1
# name            <active_objs> <num_objs> <objsize> <objperslab> <pagesperslab> : tunables <limit> <batchcount> <sharedfactor> : slabdata <active_slabs> <num_slabs> <sharedavail>
ext4_groupinfo_1k     32     32    128   32    1 : tunables    0    0    0 : slabdata      1      1      0
RAWv6                 34     34    960   34    8 : tunables    0    0    0 : slabdata      1      1      0
UDPLITEv6              0      0    960   34    8 : tunables    0    0    0 : slabdata      0      0      0
UDPv6                544    544    960   34    8 : tunables    0    0    0 : slabdata     16     16      0
tw_sock_TCPv6          0      0    320   25    2 : tunables    0    0    0 : slabdata      0      0      0
TCPv6                 72     72   1728   18    8 : tunables    0    0    0 : slabdata      4      4      0
nf_conntrack_expect    592    592    216   37    2 : tunables    0    0    0 : slabdata     16     16      0
nf_conntrack_ffffffff8199a280    933   1856    280   29    2 : tunables    0    0    0 : slabdata     64     64      0
dm_raid1_read_record      0      0   1064   30    8 : tunables    0    0    0 : slabdata      0      0      0
dm_snap_pending_exception      0      0    104   39    1 : tunables    0    0    0 : slabdata      0      0      0
dm_crypt_io         1811   2574    152   26    1 : tunables    0    0    0 : slabdata     99     99      0
kcopyd_job             0      0   3240   10    8 : tunables    0    0    0 : slabdata      0      0      0
dm_uevent              0      0   2608   12    8 : tunables    0    0    0 : slabdata      0      0      0
cfq_queue              0      0    232   35    2 : tunables    0    0    0 : slabdata      0      0      0
bsg_cmd                0      0    312   26    2 : tunables    0    0    0 : slabdata      0      0      0
mqueue_inode_cache     36     36    896   36    8 : tunables    0    0    0 : slabdata      1      1      0
udf_inode_cache        0      0    656   24    4 : tunables    0    0    0 : slabdata      0      0      0
fuse_request           0      0    608   26    4 : tunables    0    0    0 : slabdata      0      0      0
fuse_inode             0      0    704   46    8 : tunables    0    0    0 : slabdata      0      0      0
ntfs_big_inode_cache      0      0    832   39    8 : tunables    0    0    0 : slabdata      0      0      0
ntfs_inode_cache       0      0    280   29    2 : tunables    0    0    0 : slabdata      0      0      0
isofs_inode_cache      0      0    600   27    4 : tunables    0    0    0 : slabdata      0      0      0
fat_inode_cache        0      0    664   24    4 : tunables    0    0    0 : slabdata      0      0      0
fat_cache              0      0     40  102    1 : tunables    0    0    0 : slabdata      0      0      0
hugetlbfs_inode_cache     28     28    568   28    4 : tunables    0    0    0 : slabdata      1      1      0
squashfs_inode_cache      0      0    640   25    4 : tunables    0    0    0 : slabdata      0      0      0
jbd2_journal_handle   2720   2720     24  170    1 : tunables    0    0    0 : slabdata     16     16      0
jbd2_journal_head    818   1620    112   36    1 : tunables    0    0    0 : slabdata     45     45      0
jbd2_revoke_record   2048   4096     32  128    1 : tunables    0    0    0 : slabdata     32     32      0
ext4_inode_cache    2754   5328    864   37    8 : tunables    0    0    0 : slabdata    144    144      0
ext4_xattr             0      0     88   46    1 : tunables    0    0    0 : slabdata      0      0      0
ext4_free_data      1168   2628     56   73    1 : tunables    0    0    0 : slabdata     36     36      0
ext4_allocation_context    540    540    136   30    1 : tunables    0    0    0 : slabdata     18     18      0
ext4_io_end            0      0   1128   29    8 : tunables    0    0    0 : slabdata      0      0      0
ext4_io_page         256    256     16  256    1 : tunables    0    0    0 : slabdata      1      1      0
configfs_dir_cache      0      0     88   46    1 : tunables    0    0    0 : slabdata      0      0      0
kioctx                 0      0    384   42    4 : tunables    0    0    0 : slabdata      0      0      0
inotify_inode_mark     30     30    136   30    1 : tunables    0    0    0 : slabdata      1      1      0
kvm_async_pf         448    448    144   28    1 : tunables    0    0    0 : slabdata     16     16      0
kvm_vcpu              64     94  13856    2    8 : tunables    0    0    0 : slabdata     47     47      0
UDP-Lite               0      0    768   42    8 : tunables    0    0    0 : slabdata      0      0      0
xfrm_dst_cache         0      0    448   36    4 : tunables    0    0    0 : slabdata      0      0      0
ip_fib_trie          219    219     56   73    1 : tunables    0    0    0 : slabdata      3      3      0
arp_cache            417    500    320   25    2 : tunables    0    0    0 : slabdata     20     20      0
RAW                  672    672    768   42    8 : tunables    0    0    0 : slabdata     16     16      0
UDP                  672    672    768   42    8 : tunables    0    0    0 : slabdata     16     16      0
tw_sock_TCP          512   1088    256   32    2 : tunables    0    0    0 : slabdata     34     34      0
TCP                  345    357   1536   21    8 : tunables    0    0    0 : slabdata     17     17      0
blkdev_queue         414    440   1616   20    8 : tunables    0    0    0 : slabdata     22     22      0
blkdev_requests      945   2209    344   47    4 : tunables    0    0    0 : slabdata     47     47      0
sock_inode_cache     456    475    640   25    4 : tunables    0    0    0 : slabdata     19     19      0
shmem_inode_cache   2063   2375    632   25    4 : tunables    0    0    0 : slabdata     95     95      0
Acpi-ParseExt       3848   3864     72   56    1 : tunables    0    0    0 : slabdata     69     69      0
Acpi-Namespace    633667 1059270     40  102    1 : tunables    0    0    0 : slabdata  10385  10385      0
task_delay_info     1238   1584    112   36    1 : tunables    0    0    0 : slabdata     44     44      0
taskstats            384    384    328   24    2 : tunables    0    0    0 : slabdata     16     16      0
proc_inode_cache    2460   3250    616   26    4 : tunables    0    0    0 : slabdata    125    125      0
sigqueue             400    400    160   25    1 : tunables    0    0    0 : slabdata     16     16      0
bdev_cache           701    714    768   42    8 : tunables    0    0    0 : slabdata     17     17      0
sysfs_dir_cache    31662  34425     80   51    1 : tunables    0    0    0 : slabdata    675    675      0
inode_cache         2546   3886    552   29    4 : tunables    0    0    0 : slabdata    134    134      0
dentry              9452  14868    192   42    2 : tunables    0    0    0 : slabdata    354    354      0
buffer_head       8175114 8360937    104   39    1 : tunables    0    0    0 : slabdata 214383 214383      0
vm_area_struct     35344  35834    176   46    2 : tunables    0    0    0 : slabdata    782    782      0
files_cache          736    874    704   46    8 : tunables    0    0    0 : slabdata     19     19      0
signal_cache        1011   1296    896   36    8 : tunables    0    0    0 : slabdata     36     36      0
sighand_cache        682    945   2112   15    8 : tunables    0    0    0 : slabdata     63     63      0
task_struct         1057   1386   1520   21    8 : tunables    0    0    0 : slabdata     66     66      0
anon_vma            2417   2856     72   56    1 : tunables    0    0    0 : slabdata     51     51      0
shared_policy_node   4877   6800     48   85    1 : tunables    0    0    0 : slabdata     80     80      0
numa_policy        45589  48450     24  170    1 : tunables    0    0    0 : slabdata    285    285      0
radix_tree_node   227192 248388    568   28    4 : tunables    0    0    0 : slabdata   9174   9174      0
idr_layer_cache      603    660    544   30    4 : tunables    0    0    0 : slabdata     22     22      0
dma-kmalloc-8192       0      0   8192    4    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-4096       0      0   4096    8    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-2048       0      0   2048   16    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-1024       0      0   1024   32    8 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-512        0      0    512   32    4 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-256        0      0    256   32    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-128        0      0    128   32    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-64         0      0     64   64    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-32         0      0     32  128    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-16         0      0     16  256    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-8          0      0      8  512    1 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-192        0      0    192   42    2 : tunables    0    0    0 : slabdata      0      0      0
dma-kmalloc-96         0      0     96   42    1 : tunables    0    0    0 : slabdata      0      0      0
kmalloc-8192          88    100   8192    4    8 : tunables    0    0    0 : slabdata     25     25      0
kmalloc-4096        3567   3704   4096    8    8 : tunables    0    0    0 : slabdata    463    463      0
kmalloc-2048       55140  55936   2048   16    8 : tunables    0    0    0 : slabdata   3496   3496      0
kmalloc-1024        5960   6496   1024   32    8 : tunables    0    0    0 : slabdata    203    203      0
kmalloc-512        12185  12704    512   32    4 : tunables    0    0    0 : slabdata    397    397      0
kmalloc-256       195078 199040    256   32    2 : tunables    0    0    0 : slabdata   6220   6220      0
kmalloc-128        45645  47328    128   32    1 : tunables    0    0    0 : slabdata   1479   1479      0
kmalloc-64        14647251 14776576     64   64    1 : tunables    0    0    0 : slabdata 230884 230884      0
kmalloc-32          5573   7552     32  128    1 : tunables    0    0    0 : slabdata     59     59      0
kmalloc-16          7550  10752     16  256    1 : tunables    0    0    0 : slabdata     42     42      0
kmalloc-8          13805  14848      8  512    1 : tunables    0    0    0 : slabdata     29     29      0
kmalloc-192        47641  50883    192   42    2 : tunables    0    0    0 : slabdata   1214   1214      0
kmalloc-96          3673   6006     96   42    1 : tunables    0    0    0 : slabdata    143    143      0
kmem_cache            32     32    256   32    2 : tunables    0    0    0 : slabdata      1      1      0
kmem_cache_node      495    576     64   64    1 : tunables    0    0    0 : slabdata      9      9      0

# cat /proc/buddyinfo
Node 0, zone      DMA      0      0      1      0      2      1      1      0      1      1      3
Node 0, zone    DMA32   9148   1941    657    673    131     53     18      2      0      0      0
Node 0, zone   Normal   8080     13      0      2      0      2      1      0      1      0      0
Node 1, zone   Normal  19071   3239    675    200    413     37      4      1      2      0      0
Node 2, zone   Normal  37716   3924    154      9      3      1      2      0      1      0      0
Node 3, zone   Normal  20015   4590   1768    996    334     20      1      1      1      0      0

# grep MemFree /sys/devices/system/node/node*/meminfo
/sys/devices/system/node/node0/meminfo:Node 0 MemFree:          201460 kB
/sys/devices/system/node/node1/meminfo:Node 1 MemFree:          283224 kB
/sys/devices/system/node/node2/meminfo:Node 2 MemFree:          287060 kB
/sys/devices/system/node/node3/meminfo:Node 3 MemFree:          316928 kB

# cat /proc/vmstat
nr_free_pages 224933
nr_inactive_anon 1173838
nr_active_anon 5209232
nr_inactive_file 6998686
nr_active_file 2180311
nr_unevictable 1685
nr_mlock 1685
nr_anon_pages 5940145
nr_mapped 1836
nr_file_pages 9635092
nr_dirty 603
nr_writeback 0
nr_slab_reclaimable 253121
nr_slab_unreclaimable 299440
nr_page_table_pages 32311
nr_kernel_stack 854
nr_unstable 0
nr_bounce 0
nr_vmscan_write 50485772
nr_writeback_temp 0
nr_isolated_anon 0
nr_isolated_file 0
nr_shmem 12
nr_dirtied 5630347228
nr_written 5625041387
numa_hit 28372623283
numa_miss 4761673976
numa_foreign 4761673976
numa_interleave 30490
numa_local 28372334279
numa_other 4761962980
nr_anon_transparent_hugepages 76
nr_dirty_threshold 8192
nr_dirty_background_threshold 4096
pgpgin 9523143630
pgpgout 23124688920
pswpin 57978726
pswpout 50121412
pgalloc_dma 0
pgalloc_dma32 1132547190
pgalloc_normal 32421613044
pgalloc_movable 0
pgfree 39379011152
pgactivate 751722445
pgdeactivate 591205976
pgfault 41103638391
pgmajfault 11853858
pgrefill_dma 0
pgrefill_dma32 24124080
pgrefill_normal 540719764
pgrefill_movable 0
pgsteal_dma 0
pgsteal_dma32 297677595
pgsteal_normal 4784595717
pgsteal_movable 0
pgscan_kswapd_dma 0
pgscan_kswapd_dma32 241277864
pgscan_kswapd_normal 4004618399
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 65729843
pgscan_direct_normal 1012932822
pgscan_direct_movable 0
zone_reclaim_failed 0
pginodesteal 66
slabs_scanned 668153728
kswapd_steal 4063341017
kswapd_inodesteal 2063
kswapd_low_wmark_hit_quickly 9834
kswapd_high_wmark_hit_quickly 488468
kswapd_skip_congestion_wait 580150
pageoutrun 22006623
allocstall 926752
pgrotated 28467920
compact_blocks_moved 522323130
compact_pages_moved 5774251432
compact_pagemigrate_failed 5267247
compact_stall 121045
compact_fail 68349
compact_success 52696
htlb_buddy_alloc_success 0
htlb_buddy_alloc_fail 0
unevictable_pgs_culled 19976952
unevictable_pgs_scanned 0
unevictable_pgs_rescued 33137561
unevictable_pgs_mlocked 35042070
unevictable_pgs_munlocked 33138335
unevictable_pgs_cleared 0
unevictable_pgs_stranded 0
unevictable_pgs_mlockfreed 1024
thp_fault_alloc 263176
thp_fault_fallback 717335
thp_collapse_alloc 21307
thp_collapse_alloc_failed 91103
thp_split 90328

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@xxxxxxxxx.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>


[Index of Archives]     [Linux ARM Kernel]     [Linux ARM]     [Linux Omap]     [Fedora ARM]     [IETF Annouce]     [Bugtraq]     [Linux]     [Linux OMAP]     [Linux MIPS]     [ECOS]     [Asterisk Internet PBX]     [Linux API]