So, I started anew. I decreased the memory allocated to each guest to 3500MB (from 3500MB as I told earlier), but have not decreased number of hugepages - it is still 3696. On one host I started one guest. it looked like this: HugePages_Total: 3696 HugePages_Free: 1933 HugePages_Rsvd: 19 HugePages_Surp: 0 Hugepagesize: 2048 kB top - 22:05:53 up 2 days, 3:44, 1 user, load average: 0.29, 0.33, 0.29 Tasks: 131 total, 1 running, 130 sleeping, 0 stopped, 0 zombie Cpu(s): 0.9%us, 4.6%sy, 0.0%ni, 90.8%id, 1.0%wa, 0.0%hi, 2.7%si, 0.0%st Mem: 8193472k total, 8118248k used, 75224k free, 29036k buffers Swap: 11716412k total, 0k used, 11716412k free, 75864k cached procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 74668 29036 75864 0 0 1 8 54 51 1 7 91 1 Now I am starting the second virtual machine, and that's what happens: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 0 74272 29216 76664 0 0 0 0 447 961 0 0 100 0 0 0 0 73172 29216 77464 0 0 192 16 899 1575 1 1 96 2 0 0 0 72528 29224 77464 0 0 0 14 475 1022 1 0 99 0 0 0 0 72720 29232 77456 0 0 0 49 519 999 0 0 97 3 1 0 52 77988 28776 40492 0 10 1191 17 988 2285 8 9 72 11 4 0 52 68868 28784 40492 0 0 2854 38 7452 2817 17 16 67 1 2 0 52 66052 28784 40984 0 0 1906 18 24057 4620 25 20 48 7 1 0 52 67044 28792 40984 0 0 1630 35 3175 3966 9 12 72 7 0 0 52 63684 28800 40980 0 0 1433 228 6021 4479 10 11 65 14 0 1 52 65516 28800 40984 0 0 1288 109 4143 4179 10 10 58 21 2 2 52 62216 28808 40984 0 0 1698 241 4357 4183 9 8 58 25 2 2 52 60292 28816 40984 0 0 2874 258 11538 5324 15 14 39 33 2 2 52 57352 28816 40984 0 0 5303 278 8528 5176 9 11 39 42 0 7 52 54000 28824 40980 0 0 5263 249 10580 6214 16 10 32 42 0 4 396 55180 19740 40188 0 70 10304 315 7359 9633 19 8 44 28 1 0 320 61520 19748 40480 0 0 5361 302 2509 5743 23 2 50 25 1 5 316 59940 19748 40728 0 0 2343 8 2225 4690 13 3 75 10 3 1 316 55616 19748 40728 0 0 4435 215 7660 6057 15 6 51 28 0 16 2528 53596 17392 38468 0 529 832 834 6600 4675 8 5 11 76 3 0 2404 56176 17392 38480 1 0 6530 301 8371 5646 20 7 14 59 2 5 7480 58012 14836 33720 13 1082 3666 3155 12290 7752 17 10 20 54 2 1 7340 59628 14836 33884 0 0 5550 690 9513 7258 13 9 38 41 2 1 7288 59124 14844 34472 0 0 1524 481 4597 4688 5 6 58 31 0 3 7284 58848 14844 34472 0 0 1365 364 2171 3813 3 2 58 38 0 1 7056 59324 14844 34472 7 0 841 372 2159 3940 3 2 48 47 0 30 7056 54456 14844 34472 0 0 2 248 1402 2705 2 1 85 13 0 1 6892 55336 14828 38396 1 0 888 268 1927 4124 2 2 41 55 0 0 6892 57808 14060 36988 0 0 17 92 948 1682 1 1 93 5 0 0 6888 58616 14060 37696 0 0 140 43 747 1566 1 1 94 5 1 0 6884 59444 14060 37696 0 0 7 14 942 1747 3 1 95 1 1 0 6884 58820 14060 37696 0 0 0 46 722 1480 1 1 97 2 0 0 6884 58608 14060 37696 0 0 0 41 858 1564 3 1 93 3 3 8 6884 51752 14060 37792 0 0 354 147 8243 2447 20 7 71 2 2 0 6880 52840 14060 37792 0 0 604 281 10430 5859 21 15 50 14 0 0 6880 55176 14060 37792 0 0 699 232 3271 3656 20 4 66 10 0 0 6880 56120 14060 37792 0 0 0 280 1064 2116 1 1 85 14 0 0 6880 55628 14060 37792 0 0 0 0 616 1367 1 0 98 0 1 0 6880 56388 14060 37792 0 0 0 18 689 1381 1 1 97 2 Unlike I have expected given that in the previous case I had only 6 unreserved pages, and I thought I would have 56 now, I have 156 free unreserved pages: HugePages_Total: 3696 HugePages_Free: 1113 HugePages_Rsvd: 957 HugePages_Surp: 0 Then at one moment both guests almost stopped working for a minute or so - both went up to huge load and became unresponsive. I didn't get to catch hot they looked in 'top', but they did not use any swap themselves (they have at least 1GB of free memory each) and their load average went to something like 10. vmstat from the host looked like this: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 6740 61948 11140 34104 0 0 0 3 663 1435 1 0 99 0 0 3 6740 62796 11140 34104 0 0 0 10 992 1365 1 1 97 1 0 0 6740 62788 11140 34104 0 0 0 35 854 1647 3 1 95 0 7 0 6740 57664 11140 34104 0 0 0 238 7011 1918 34 6 57 3 9 0 6740 58212 11140 34104 0 0 0 66 3107 2065 99 1 0 0 8 0 6740 58120 11140 34104 0 0 0 61 2509 1770 100 0 0 0 8 0 6740 57624 11140 34104 0 0 0 69 2610 1955 100 0 0 0 7 0 6740 57996 11140 34104 0 0 0 10 2385 2035 100 0 0 0 7 0 6740 58348 11140 34104 0 0 0 26 2580 2296 99 1 0 0 7 0 6740 58348 11140 34104 0 0 0 10 2477 2132 100 0 0 0 8 0 6740 58580 11148 34104 0 0 0 107 2687 2221 99 1 0 0 8 0 6740 58364 11148 34104 0 0 0 20 2461 1706 100 0 0 0 8 0 6740 58504 11148 34104 0 0 0 3 2301 1678 100 0 0 0 8 0 6740 58464 11148 34104 0 0 0 32 2355 1709 100 0 0 0 8 0 6740 58704 11148 34104 0 0 0 24 2380 1749 100 0 0 0 8 0 6740 58820 11148 34104 0 0 0 26 2340 1630 100 0 0 0 8 0 6740 59208 11148 34104 0 0 0 41 2360 1711 99 1 0 0 8 0 6740 58952 11148 34104 0 0 0 22 2387 1648 99 1 0 0 8 0 6740 59184 11148 34104 0 0 0 0 2315 1690 100 1 0 0 8 0 6740 59332 11148 34104 0 0 0 0 2383 1637 100 0 0 0 8 0 6740 59084 11148 34104 0 0 0 0 2357 1692 100 0 0 0 8 0 6740 58472 11148 34104 0 0 0 45 2497 1790 96 4 0 0 8 0 6740 58828 11156 34104 0 0 1 56 2483 1788 89 11 0 0 8 0 6740 58852 11156 34104 0 0 0 27 2385 1707 95 5 0 0 8 0 6740 59240 11156 34104 0 0 0 20 2387 1735 97 3 0 0 8 0 6740 59216 11156 34104 0 0 1 49 2392 1698 99 1 0 0 8 0 6740 58936 11156 34104 0 0 0 0 2327 1678 99 1 0 0 8 0 6740 59092 11156 34104 0 0 0 63 2490 1732 99 1 0 0 9 0 6740 58688 11156 34104 0 0 0 106 2487 1826 99 1 0 0 8 0 6740 58596 11156 34104 0 0 0 87 2509 1839 99 1 0 0 10 0 6740 58524 11156 34104 0 0 0 61 2550 1839 100 0 0 0 8 0 6740 58820 11156 34104 0 0 0 66 2450 1796 99 1 0 0 8 0 6740 56712 11156 34104 0 0 0 52 4595 1721 99 1 0 0 2 0 7672 68052 6924 28256 2 190 568 306 14201 2760 70 11 15 4 0 0 7668 69880 6932 28428 2 0 407 355 2387 3029 13 2 78 7 0 1 7632 68056 6940 29836 0 0 283 221 1016 1801 1 1 90 9 0 0 7632 68200 6940 29836 0 0 1 195 1165 1963 1 1 88 11 0 3 7564 66728 6940 29904 0 0 0 100 885 1788 1 1 89 9 0 0 7560 69176 6940 29840 0 0 0 19 647 1327 0 1 98 1 A few minutes later it happened again, but for longer time (I waited for at least 5 minutes, and it didn't get any better). vmstat showed this: procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 7528 67192 7080 31920 0 0 0 4 658 1401 1 0 99 0 0 0 7528 67264 7080 31920 0 0 0 16 867 1555 4 1 96 0 9 0 7528 62304 7080 31920 0 0 0 153 7415 1852 37 4 58 1 8 4 7528 62132 7080 31920 0 0 0 78 2743 1821 100 0 0 0 8 0 7528 61488 7080 31920 0 0 0 79 2854 1833 100 0 0 0 9 0 7528 61448 7080 31920 0 0 0 48 2628 1698 100 0 0 0 8 0 7528 61680 7080 31920 0 0 0 24 2643 1733 100 1 0 0 8 0 7528 61572 7080 31920 0 0 0 26 2488 1661 100 0 0 0 8 0 7528 61168 7080 31920 0 0 0 50 2493 1804 100 0 0 0 8 0 7528 61432 7088 31920 0 0 0 48 2461 1669 100 0 0 0 8 0 7528 61852 7088 31920 0 0 0 52 2529 1841 100 1 0 0 8 0 7528 62132 7088 31920 0 0 0 39 2471 1642 100 1 0 0 8 0 7528 61404 7088 31920 0 0 0 26 2468 1722 100 0 0 0 8 0 7528 61256 7088 31920 0 0 0 27 2457 1644 100 0 0 0 8 0 7528 61908 7088 31920 0 0 0 46 2477 1749 100 0 0 0 8 0 7528 61000 7088 31920 0 0 0 46 2954 1675 99 1 0 0 8 0 7528 60000 7088 31920 0 0 1 67 2834 1786 99 1 0 0 8 0 7528 59504 7088 31920 0 0 253 26 2410 1618 100 0 0 0 8 0 7528 59620 7088 31920 0 0 34 83 2631 1791 100 0 0 0 8 0 7528 59728 7088 31920 0 0 29 58 2531 1646 100 0 0 0 8 0 7528 59752 7088 31920 0 0 0 29 2517 1787 99 1 0 0 8 0 7528 59620 7096 31920 0 0 3 52 2448 1648 99 1 0 0 8 0 7528 58992 7096 31920 0 0 0 45 2536 1745 100 0 0 0 9 1 7528 59024 7096 31920 0 0 1 38 2548 1635 99 1 0 0 8 0 7528 58768 7096 31920 0 0 0 44 2496 1741 100 0 0 0 8 0 7528 59388 7096 31920 0 0 0 18 2429 1617 100 0 0 0 8 0 7528 58868 7096 31920 0 0 0 51 2600 1745 100 0 0 0 8 0 7528 59156 7104 31920 0 0 0 47 2441 1682 100 0 0 0 9 0 7528 57380 7104 31920 0 0 0 40 2709 1690 100 1 0 0 8 0 7528 58056 7104 31920 0 0 1 8 3127 1629 100 0 0 0 9 0 7528 57544 7104 31920 0 0 0 36 2615 1704 100 0 0 0 8 0 7528 57196 7104 31920 0 0 0 26 2530 1710 100 0 0 0 8 0 7528 59792 6836 28648 0 0 0 42 2613 1761 100 0 0 0 8 0 7528 59156 6844 29036 0 0 78 64 2641 1757 100 0 0 0 8 0 7528 59576 6844 29164 0 0 26 26 2462 1632 100 0 0 0 8 0 7528 59716 6844 29164 0 0 0 8 2414 1706 100 0 0 0 8 0 7528 59600 6844 29164 0 0 0 48 2505 1649 100 0 0 0 8 0 7528 59600 6844 29164 0 0 0 9 2373 1648 99 1 0 0 8 0 7528 59492 6844 29164 0 0 0 10 2387 1564 100 1 0 0 8 0 7528 59624 6844 29164 0 0 0 40 2551 1691 100 0 0 0 8 0 7528 59080 6844 29164 0 0 0 50 2733 1643 100 0 0 0 8 0 7528 58956 6844 29164 0 0 0 29 2823 1652 100 0 0 0 8 1 7528 58624 6844 29164 0 0 0 34 2478 1633 100 0 0 0 8 0 7528 58716 6844 29164 0 0 0 39 2398 1688 99 1 0 0 8 0 7528 57592 6844 30228 0 0 212 30 2373 1666 100 1 0 0 8 0 7528 57468 6852 30228 0 0 0 51 2453 1695 100 0 0 0 8 0 7528 58244 6852 30228 0 0 0 26 2756 1617 99 1 0 0 8 0 7528 58244 6852 30228 0 0 0 112 3872 1952 99 1 0 0 9 1 7528 58320 6852 30228 0 0 0 48 2718 1719 100 0 0 0 8 0 7528 58204 6852 30228 0 0 0 17 2692 1697 100 0 0 0 8 0 7528 59220 6852 30228 0 0 5 48 4666 1651 98 2 0 0 9 0 7528 57716 6852 30228 0 0 0 101 5128 1874 98 2 0 0 9 0 7528 55692 6860 30228 0 0 5 100 5875 1825 97 3 0 0 9 1 7528 55668 6860 30228 0 0 0 156 3910 1960 99 1 0 0 8 0 7528 55668 6860 30228 0 0 0 38 2578 1671 100 0 0 0 8 0 7528 55600 6860 30228 0 0 2 81 2783 1888 100 0 0 0 9 1 7528 59660 5188 28320 0 0 0 50 2601 1918 100 0 0 0 8 0 7528 63280 5196 28328 0 0 2 63 4347 1855 99 1 0 0 8 0 7528 62560 5196 28328 0 0 0 101 3383 1748 99 1 0 0 9 0 7528 62132 5196 28328 0 0 1 50 2656 1724 100 0 0 0 One guest showed this: top - 23:11:35 up 53 min, 1 user, load average: 20.42, 14.61, 7.26 Tasks: 205 total, 40 running, 165 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 99.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 3510920k total, 972876k used, 2538044k free, 30256k buffers Swap: 4194296k total, 0k used, 4194296k free, 321288k cached The other one: top - 23:11:12 up 1 day, 9:19, 1 user, load average: 19.38, 14.54, 7.40 Tasks: 219 total, 15 running, 204 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 99.6%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.1%si, 0.0%st Mem: 3510920k total, 1758688k used, 1752232k free, 298000k buffers Swap: 4194296k total, 0k used, 4194296k free, 577068k cached Thanks, Dmitry On Sun, Oct 3, 2010 at 12:28 PM, Avi Kivity <avi@xxxxxxxxxx> wrote: > On 09/30/2010 11:07 AM, Dmitry Golubev wrote: >> >> Hi, >> >> I am not sure what's really happening, but every few hours >> (unpredictable) two virtual machines (Linux 2.6.32) start to generate >> huge cpu loads. It looks like some kind of loop is unable to complete >> or something... >> > > What does 'top' inside the guest show when this is happening? > > -- > error compiling committee.c: too many arguments to function > > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
