* Chris Webb <chris@xxxxxxxxxxxx> [2010-08-02 13:47:35]: > We run a number of relatively large x86-64 hosts with twenty or so qemu-kvm > virtual machines on each of them, and I'm have some trouble with over-eager > swapping on some (but not all) of the machines. This is resulting in > customer reports of very poor response latency from the virtual machines > which have been swapped out, despite the hosts apparently having large > amounts of free memory, and running fine if swap is turned off. > > All of the hosts are running a 2.6.32.7 kernel and have ksm enabled with > 32GB of RAM and 2x quad-core processors. There is a cluster of Xeon E5420 > machines which apparently doesn't exhibit the problem, and a cluster of > 2352/2378 Opteron (NUMA) machines, some of which do. The kernel config of > the affected machines is at > > http://cdw.me.uk/tmp/config-2.6.32.7 > > This differs very little from the config on the unaffected Xeon machines, > essentially just > > -CONFIG_MCORE2=y > +CONFIG_MK8=y > -CONFIG_X86_P6_NOP=y > > On a typical affected machine, the virtual machines and other processes > would apparently leave around 5.5GB of RAM available for buffers, but the > system seems to want to swap out 3GB of anonymous pages to give itself more > like 9GB of buffers: > > # cat /proc/meminfo > MemTotal: 33083420 kB > MemFree: 693164 kB > Buffers: 8834380 kB > Cached: 11212 kB > SwapCached: 1443524 kB > Active: 21656844 kB > Inactive: 8119352 kB > Active(anon): 17203092 kB > Inactive(anon): 3729032 kB > Active(file): 4453752 kB > Inactive(file): 4390320 kB > Unevictable: 5472 kB > Mlocked: 5472 kB > SwapTotal: 25165816 kB > SwapFree: 21854572 kB > Dirty: 4300 kB > Writeback: 4 kB > AnonPages: 20780368 kB > Mapped: 6056 kB > Shmem: 56 kB > Slab: 961512 kB > SReclaimable: 438276 kB > SUnreclaim: 523236 kB > KernelStack: 10152 kB > PageTables: 67176 kB > NFS_Unstable: 0 kB > Bounce: 0 kB > WritebackTmp: 0 kB > CommitLimit: 41707524 kB > Committed_AS: 39870868 kB > VmallocTotal: 34359738367 kB > VmallocUsed: 150880 kB > VmallocChunk: 34342404996 kB > HardwareCorrupted: 0 kB > HugePages_Total: 0 > HugePages_Free: 0 > HugePages_Rsvd: 0 > HugePages_Surp: 0 > Hugepagesize: 2048 kB > DirectMap4k: 5824 kB > DirectMap2M: 3205120 kB > DirectMap1G: 30408704 kB > > We see this despite the machine having vm.swappiness set to 0 in an attempt > to skew the reclaim as far as possible in favour of releasing page cache > instead of swapping anonymous pages. > > After running swapoff -a, the machine is immediately much healthier. Even > while the swap is still being reduced, load goes down and response times in > virtual machines are much improved. Once the swap is completely gone, there > are still several gigabytes of RAM left free which are used for buffers, and > the virtual machines are no longer laggy because they are no longer swapped > out. Running swapon -a again, the affected machine waits for about a minute > with zero swap in use, before the amount of swap in use very rapidly > increases to around 2GB and then continues to increase more steadily to 3GB. > > We could run with these machines without swap (in the worst cases we're > already doing so), but I'd prefer to have a reserve of swap available in > case of genuine emergency. If it's a choice between swapping out a guest or > oom-killing it, I'd prefer to swap... but I really don't want to swap out > running virtual machines in order to have eight gigabytes of page cache > instead of five! > > Is this a problem with the page reclaim priorities, or am I just tuning > these hosts incorrectly? Is there more detailed info than /proc/meminfo > available which might shed more light on what's going wrong here? > Can you give an idea of what the meminfo inside the guest looks like. Have you looked at http://kerneltrap.org/mailarchive/linux-kernel/2010/6/8/4580772 -- Three Cheers, Balbir -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@xxxxxxxxxx For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@xxxxxxxxx";> email@xxxxxxxxx </a>