Hi to be as verbose and clear as possible, I will write every command I use. This sets everything up for the test: # setup start importantvms="kernel1 kernel2" notimportantvms="kernel3 kernel4 kernel5 kernel6 kernel7 kernel8" for vm in $importantvms $notimportantvms; do virsh domstate $vm|grep -q running || virsh start $vm; done echo 1 > /sys/block/sdb/queue/iosched/group_isolation mount -t cgroup -o blkio none /mnt cd /mnt mkdir important mkdir notimportant for vm in $notimportantvms; do cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task for task in *; do /bin/echo $task > /mnt/notimportant/tasks done done for vm in $importantvms; do cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task for task in *; do /bin/echo $task > /mnt/important/tasks done done #ls -lH /dev/vdisks/kernel[3-8] #brw-rw---- 1 root root 254, 25 Feb 22 13:42 kernel3 #brw-rw---- 1 root root 254, 26 Feb 22 13:42 kernel4 #brw-rw---- 1 root root 254, 27 Feb 22 13:42 kernel5 #brw-rw---- 1 root root 254, 28 Feb 22 13:42 kernel6 #brw-rw---- 1 root root 254, 29 Feb 22 13:43 kernel7 #brw-rw---- 1 root root 254, 30 Feb 22 13:43 kernel8 for i in $(seq 25 30); do /bin/echo 254:$i 10000000 > /mnt/notimportant/blkio.throttle.write_bps_device done # setup complete > Hm..., this sounds bad. If you have put a limit of ~10Mb/s then no > "bo" is bad. That would explain that why your box is not responding > and you need to do power reset. > > - I am assuming that you have not put any throttling limits on root group. > Is your system root also on /dev/sdb or on a separate disk altogether. No throttling on root. Correct. system root is on sda vms are on sdb > - This sounds like a bug in throttling logic. To narrow it down can you > start running "deadline" on end device. If it still happens, it is more > or less in throttling layer. cat /sys/block/sdb/queue/scheduler noop deadline [cfq] echo deadline > /sys/block/sdb/queue/scheduler cat /sys/block/sdb/queue/scheduler noop [deadline] cfq This changes things: vmstat before test (nothing going on): r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 130231696 17968 69424 0 0 0 0 16573 32834 0 0 100 0 vmstat during test while all 8 vms are writing (2 unthrottled, 6 at 10M): r b swpd free buff cache si so bi bo in cs us sy id wa 2 9 0 126257984 17968 69424 0 0 8 164462 20114 36751 5 4 52 39 vmstat during test when only throttled vms are still dd'ing 2 21 0 124518928 17976 69424 0 0 0 63876 17410 33670 1 1 59 39 Load is 12-ish during the test and the results are as expected: Throttled VMs write with approx 10M, unthrottled get like 80 each, which sums up to about the 200M max capacity of the device for direct io. > - We can also try to remove dm layers and just create partitions on > /dev/sdb and export as virtio disks to virtual machines and take > dm layer out of picture and see if it still happens. Did that. Also happens when scheduler is cfq. Also does not happen when scheduler is deadline. So it does not seem to be a dm issue. > - In one of the mails you mentioned that with 1 virutal machine throttling > READs and WRITEs is working for you. So it looks like 1 virtual machine > does not hang but once you launch 8 virtual machines it hangs. Can we > try increasing the number of vitual machines gragually and confirm that > it happens only if some certain number of virtual machines are launched. Back on cfq here. 1 throttled vm: works, load ~4 2 throttled vms: works, load ~6 3 throttled vms: works, load ~9 4 throttled vms: works, load ~12 6 throttled vms: works, load ~20 The number of blocked threads increases with the number of vms dd'ing. At the beginning of each test, the blocked threads number goes really high (4 vms 160, 6 vms 220), but then drops significantly and stays low. So it seems that when only throttled vms are running, the problem does not occur. 1 throttled + 1 unthrottled vm: works, load ~5 2 throttled + 1 unthrottled vm: boom Constantly 144 blocked threads, bo=0, load increasing to 144. System needs power reset. So, thinking about what I did in the initial setup, I re-tested without the for vm in $importantvms; do cd /proc/$(pgrep -f "qemu-kvm.*$vm")/task for task in *; do /bin/echo $task > /mnt/important/tasks done done since I don't do anything with that "important" cgroup (yet) anyway. It did not make a difference though. > - Can you also paste me the rules you have put on important and non-important > groups. Somehow I suspect that some of the rule has gone horribly bad > in the sense that it is very low and effectively no virtual machine > is making any progress. See setup commands in the beginning of this email. > - How long does it take to reach in this locked state where bo=0. It goes there "instantly", right after the dd commands start. > - you can also try to redirect blktrace output to blkparse and redirect > it to standard output and see capture some output by copying pasting > last messages. I hope this is what you meant: blktrace -d /dev/sdb -o - | blkparse -i - Output is in attachment. Regards Dominik
8,16 3 3927 55.798466536 3693 A WS 3407889504 + 480 <- (8,17) 3407887456 8,16 3 3928 55.798467619 3693 Q WS 3407889504 + 480 [qemu-kvm] 8,16 3 3929 55.798470220 3693 G WS 3407889504 + 480 [qemu-kvm] 8,16 3 3930 55.798471438 3693 I W 3407889504 + 480 [qemu-kvm] 8,16 3 0 55.798471876 0 m N cfq3693 insert_request 8,17 3 3931 55.798491328 3693 A WS 3407887936 + 480 <- (254,23) 33569984 8,16 3 3932 55.798491556 3693 A WS 3407889984 + 480 <- (8,17) 3407887936 8,16 3 3933 55.798491904 3693 Q WS 3407889984 + 480 [qemu-kvm] 8,16 3 3934 55.798492752 3693 G WS 3407889984 + 480 [qemu-kvm] 8,16 3 3935 55.798493994 3693 I W 3407889984 + 480 [qemu-kvm] 8,16 3 0 55.798494516 0 m N cfq3693 insert_request 8,17 3 3936 55.798510421 3693 A WS 3407888416 + 480 <- (254,23) 33570464 8,16 3 3937 55.798510610 3693 A WS 3407890464 + 480 <- (8,17) 3407888416 8,16 3 3938 55.798510926 3693 Q WS 3407890464 + 480 [qemu-kvm] 8,16 3 3939 55.798511583 3693 G WS 3407890464 + 480 [qemu-kvm] 8,16 3 3940 55.798512645 3693 I W 3407890464 + 480 [qemu-kvm] 8,16 3 0 55.798518244 0 m N cfq3693 insert_request 8,17 3 3941 55.798535943 3693 A WS 3407888896 + 480 <- (254,23) 33570944 8,16 3 3942 55.798536140 3693 A WS 3407890944 + 480 <- (8,17) 3407888896 8,16 3 3943 55.798536454 3693 Q WS 3407890944 + 480 [qemu-kvm] 8,16 3 3944 55.798537257 3693 G WS 3407890944 + 480 [qemu-kvm] 8,16 3 3945 55.798538417 3693 I W 3407890944 + 480 [qemu-kvm] 8,16 3 0 55.798538844 0 m N cfq3693 insert_request 8,16 9 404 55.798556926 3564 U N [qemu-kvm] 132 8,17 3 3946 55.798557461 3693 A WS 3407889376 + 480 <- (254,23) 33571424 8,16 3 3947 55.798557645 3693 A WS 3407891424 + 480 <- (8,17) 3407889376 8,16 3 3948 55.798557990 3693 Q WS 3407891424 + 480 [qemu-kvm] 8,16 3 3949 55.798558736 3693 G WS 3407891424 + 480 [qemu-kvm] 8,16 3 3950 55.798560030 3693 I W 3407891424 + 480 [qemu-kvm] 8,16 3 0 55.798560712 0 m N cfq3693 insert_request 8,17 3 3951 55.798576762 3693 A WS 3407889856 + 152 <- (254,23) 33571904 8,16 3 3952 55.798577074 3693 A WS 3407891904 + 152 <- (8,17) 3407889856 8,16 3 3953 55.798577380 3693 Q WS 3407891904 + 152 [qemu-kvm] 8,16 3 3954 55.798578078 3693 G WS 3407891904 + 152 [qemu-kvm] 8,16 3 3955 55.798578734 3693 I W 3407891904 + 152 [qemu-kvm] 8,16 3 0 55.798579207 0 m N cfq3693 insert_request 8,16 3 3956 55.798580865 3693 U N [qemu-kvm] 134 8,17 19 950 55.798590132 3698 A WS 3407891072 + 480 <- (254,23) 33573120 8,16 19 951 55.798591023 3698 A WS 3407893120 + 480 <- (8,17) 3407891072 8,16 19 952 55.798592021 3698 Q WS 3407893120 + 480 [qemu-kvm] 8,16 19 953 55.798600061 3698 S WS 3407893120 + 480 [qemu-kvm] 8,16 6 688 55.799223015 3699 U N [qemu-kvm] 134 8,16 6 689 55.799819390 3700 U N [qemu-kvm] 134 8,16 6 690 55.800236635 3701 U N [qemu-kvm] 134 8,16 7 880 55.800279272 3702 U N [qemu-kvm] 134 8,16 6 691 55.800831081 3703 U N [qemu-kvm] 134 8,16 6 692 55.801516229 3704 U N [qemu-kvm] 134 8,16 6 693 55.801860219 3705 U N [qemu-kvm] 134 8,16 6 694 55.802110337 3706 U N [qemu-kvm] 134 8,16 6 695 55.804130110 3707 U N [qemu-kvm] 134 8,16 1 0 55.806298452 0 m N cfq idle timer fired 8,16 1 0 55.806300924 0 m N cfq3564 slice expired t=0 8,16 1 0 55.806303380 0 m N cfq3564 sl_used=2 disp=1 charge=2 iops=0 sect=8 8,16 1 0 55.806303770 0 m N cfq3564 del_from_rr 8,16 1 0 55.806304555 0 m N cfq schedule dispatch 8,17 9 405 55.806320403 484 A WS 3659277360 + 480 <- (254,29) 33301168 8,16 9 406 55.806321212 484 A WS 3659279408 + 480 <- (8,17) 3659277360 8,17 1 13713 55.806326506 454 A WS 3701251800 + 480 <- (254,30) 33332568 8,16 1 13714 55.806327201 454 A WS 3701253848 + 480 <- (8,17) 3701251800 8,16 9 407 55.806328664 484 Q WS 3659279408 + 480 [kworker/9:1] 8,16 1 13715 55.806335493 454 Q WS 3701253848 + 480 [kworker/1:1] 8,17 19 954 55.806340403 520 A WS 3533640576 + 480 <- (254,26) 33493504 8,16 9 408 55.806340854 484 S WS 3659279408 + 480 [kworker/9:1] 8,16 19 955 55.806341348 520 A WS 3533642624 + 480 <- (8,17) 3533640576 8,16 1 13716 55.806347196 454 S WS 3701253848 + 480 [kworker/1:1] 8,16 19 956 55.806356670 520 Q WS 3533642624 + 480 [kworker/19:1] 8,16 19 957 55.806364627 520 S WS 3533642624 + 480 [kworker/19:1] 8,17 14 3809 55.808056903 3708 A WS 3449578144 + 480 <- (254,24) 33317152 8,16 14 3810 55.808057746 3708 A WS 3449580192 + 480 <- (8,17) 3449578144 8,16 14 3811 55.808058575 3708 Q WS 3449580192 + 480 [qemu-kvm] 8,16 14 3812 55.808065313 3708 S WS 3449580192 + 480 [qemu-kvm] 8,16 19 958 55.808281456 3709 U N [qemu-kvm] 134 8,16 14 3813 55.808479046 3710 U N [qemu-kvm] 134 8,16 14 3814 55.808610181 3711 U N [qemu-kvm] 134 8,17 14 3815 55.808643307 3712 A WS 3449581176 + 104 <- (254,24) 33320184 8,16 14 3816 55.808643709 3712 A WS 3449583224 + 104 <- (8,17) 3449581176 8,16 14 3817 55.808644114 3712 Q WS 3449583224 + 104 [qemu-kvm] 8,16 14 3818 55.808645156 3712 S WS 3449583224 + 104 [qemu-kvm] 8,16 14 3819 55.808941624 3713 U N [qemu-kvm] 134 8,17 6 696 55.809300401 3714 A WS 3449581296 + 480 <- (254,24) 33320304 8,16 6 697 55.809301256 3714 A WS 3449583344 + 480 <- (8,17) 3449581296 8,16 6 698 55.809302071 3714 Q WS 3449583344 + 480 [qemu-kvm] 8,16 6 699 55.809307149 3714 S WS 3449583344 + 480 [qemu-kvm] 8,16 7 881 55.809415138 3715 U N [qemu-kvm] 134 8,17 14 3820 55.809630538 3717 A WS 3407893096 + 152 <- (254,23) 33575144 8,16 14 3821 55.809631395 3717 A WS 3407895144 + 152 <- (8,17) 3407893096 8,16 14 3822 55.809632498 3717 Q WS 3407895144 + 152 [qemu-kvm] 8,16 14 3823 55.809637373 3717 S WS 3407895144 + 152 [qemu-kvm] 8,16 19 959 55.809652187 3716 U N [qemu-kvm] 134 8,17 6 700 55.809726441 3718 A WS 3449583320 + 480 <- (254,24) 33322328 8,16 6 701 55.809726919 3718 A WS 3449585368 + 480 <- (8,17) 3449583320 8,16 6 702 55.809727583 3718 Q WS 3449585368 + 480 [qemu-kvm] 8,16 6 703 55.809728880 3718 S WS 3449585368 + 480 [qemu-kvm] 8,17 6 704 55.810330315 3720 A WS 3449586352 + 480 <- (254,24) 33325360 8,16 6 705 55.810330740 3720 A WS 3449588400 + 480 <- (8,17) 3449586352 8,16 6 706 55.810330979 3720 Q WS 3449588400 + 480 [qemu-kvm] 8,16 6 707 55.810331793 3720 S WS 3449588400 + 480 [qemu-kvm] 8,17 7 882 55.810342846 3719 A WS 3407893248 + 152 <- (254,23) 33575296 8,16 7 883 55.810343814 3719 A WS 3407895296 + 152 <- (8,17) 3407893248 8,16 7 884 55.810344524 3719 Q WS 3407895296 + 152 [qemu-kvm] 8,16 7 885 55.810348787 3719 S WS 3407895296 + 152 [qemu-kvm] 8,16 7 886 55.810963240 3721 U N [qemu-kvm] 134 8,17 7 887 55.811078088 3722 A WS 3449589952 + 480 <- (254,24) 33328960 8,16 7 888 55.811078908 3722 A WS 3449592000 + 480 <- (8,17) 3449589952 8,16 7 889 55.811079530 3722 Q WS 3449592000 + 480 [qemu-kvm] 8,16 7 890 55.811082649 3722 S WS 3449592000 + 480 [qemu-kvm] 8,16 7 891 55.811537529 3723 U N [qemu-kvm] 134 8,16 7 892 55.812222597 3724 U N [qemu-kvm] 134 8,16 6 708 55.812245576 3725 U N [qemu-kvm] 134 8,16 7 893 55.812274987 3726 U N [qemu-kvm] 134 8,16 6 709 55.812797561 3727 U N [qemu-kvm] 134 8,16 23 64 55.812929114 3728 U N [qemu-kvm] 134 8,16 7 894 55.813528023 3729 U N [qemu-kvm] 134 8,16 6 710 55.813687257 3730 U N [qemu-kvm] 134 8,17 7 895 55.814660300 3731 A WS 3407893408 + 480 <- (254,23) 33575456 8,16 7 896 55.814661094 3731 A WS 3407895456 + 480 <- (8,17) 3407893408 8,16 7 897 55.814661536 3731 Q WS 3407895456 + 480 [qemu-kvm] 8,16 7 898 55.814665009 3731 S WS 3407895456 + 480 [qemu-kvm] 8,16 6 711 55.814888432 3732 U N [qemu-kvm] 134 8,16 6 712 55.815519196 3733 U N [qemu-kvm] 134 8,16 6 713 55.815945610 3734 U N [qemu-kvm] 134 8,16 6 714 55.816133848 3735 U N [qemu-kvm] 134 8,16 14 3824 55.816456454 3736 U N [qemu-kvm] 134 8,16 4 1525 55.820278197 3737 U N [qemu-kvm] 134 8,16 4 1526 55.821778871 3738 U N [qemu-kvm] 134 8,16 4 1527 55.821966424 3739 U N [qemu-kvm] 134 8,16 4 1528 55.822460767 3740 U N [qemu-kvm] 134 8,16 4 1529 55.822582532 3741 U N [qemu-kvm] 134 8,16 4 1530 55.823182859 3742 U N [qemu-kvm] 134 8,17 7 899 55.823232794 3743 A WS 3407895432 + 480 <- (254,23) 33577480 8,16 7 900 55.823233549 3743 A WS 3407897480 + 480 <- (8,17) 3407895432 8,16 7 901 55.823234161 3743 Q WS 3407897480 + 480 [qemu-kvm] 8,16 7 902 55.823239257 3743 S WS 3407897480 + 480 [qemu-kvm] 8,17 7 903 55.823890130 3744 A WS 3407898464 + 480 <- (254,23) 33580512 8,16 7 904 55.823890790 3744 A WS 3407900512 + 480 <- (8,17) 3407898464 8,16 7 905 55.823891069 3744 Q WS 3407900512 + 480 [qemu-kvm] 8,16 7 906 55.823895342 3744 S WS 3407900512 + 480 [qemu-kvm] 8,17 7 907 55.824300171 3745 A WS 3407901448 + 152 <- (254,23) 33583496 8,16 7 908 55.824300596 3745 A WS 3407903496 + 152 <- (8,17) 3407901448 8,16 7 909 55.824300817 3745 Q WS 3407903496 + 152 [qemu-kvm] 8,16 7 910 55.824302076 3745 S WS 3407903496 + 152 [qemu-kvm] 8,16 4 1531 55.824325070 3746 U N [qemu-kvm] 134 8,16 19 960 55.824848417 3747 U N [qemu-kvm] 134 8,16 2 3006 55.824991838 3748 U N [qemu-kvm] 134 8,16 4 1532 55.825498152 3749 U N [qemu-kvm] 134 8,16 2 3007 55.825612338 3750 U N [qemu-kvm] 134 8,16 4 1533 55.825768633 3751 U N [qemu-kvm] 134 8,16 4 1534 55.826446820 3752 U N [qemu-kvm] 134 8,17 4 1535 55.826521140 3753 A WS 3407901608 + 480 <- (254,23) 33583656 8,16 4 1536 55.826522033 3753 A WS 3407903656 + 480 <- (8,17) 3407901608 8,16 4 1537 55.826522763 3753 Q WS 3407903656 + 480 [qemu-kvm] 8,16 4 1538 55.826527018 3753 S WS 3407903656 + 480 [qemu-kvm] 8,16 4 1539 55.827843895 3754 U N [qemu-kvm] 134 8,17 7 911 55.836641685 3755 A WS 3407903632 + 480 <- (254,23) 33585680 8,16 7 912 55.836642701 3755 A WS 3407905680 + 480 <- (8,17) 3407903632 8,16 7 913 55.836643478 3755 Q WS 3407905680 + 480 [qemu-kvm] 8,16 7 914 55.836649708 3755 S WS 3407905680 + 480 [qemu-kvm] 8,17 4 1540 55.837357705 3756 A WS 3407906664 + 480 <- (254,23) 33588712 8,16 4 1541 55.837358612 3756 A WS 3407908712 + 480 <- (8,17) 3407906664 8,16 4 1542 55.837359329 3756 Q WS 3407908712 + 480 [qemu-kvm] 8,16 4 1543 55.837363803 3756 S WS 3407908712 + 480 [qemu-kvm] 8,17 7 915 55.838540760 3757 A WS 3407907456 + 192 <- (254,23) 33589504 8,16 7 916 55.838541575 3757 A WS 3407909504 + 192 <- (8,17) 3407907456 8,16 7 917 55.838542282 3757 Q WS 3407909504 + 192 [qemu-kvm] 8,16 7 918 55.838548208 3757 S WS 3407909504 + 192 [qemu-kvm] 8,17 4 1544 55.850108060 3758 A WS 3407907840 + 480 <- (254,23) 33589888 8,16 4 1545 55.850108927 3758 A WS 3407909888 + 480 <- (8,17) 3407907840 8,16 4 1546 55.850109476 3758 Q WS 3407909888 + 480 [qemu-kvm] 8,16 4 1547 55.850113608 3758 S WS 3407909888 + 480 [qemu-kvm] 8,17 4 1548 55.850480286 3759 A WS 3407908856 + 480 <- (254,23) 33590904 8,16 4 1549 55.850480658 3759 A WS 3407910904 + 480 <- (8,17) 3407908856 8,16 4 1550 55.850480901 3759 Q WS 3407910904 + 480 [qemu-kvm] 8,16 4 1551 55.850481992 3759 S WS 3407910904 + 480 [qemu-kvm] 8,17 7 919 55.851235411 3760 A WS 3407910096 + 152 <- (254,23) 33592144 8,16 7 920 55.851236200 3760 A WS 3407912144 + 152 <- (8,17) 3407910096 8,16 7 921 55.851236889 3760 Q WS 3407912144 + 152 [qemu-kvm] 8,16 7 922 55.851243432 3760 S WS 3407912144 + 152 [qemu-kvm] 8,17 4 1552 55.854625105 3761 A WS 3407910256 + 480 <- (254,23) 33592304 8,16 4 1553 55.854625872 3761 A WS 3407912304 + 480 <- (8,17) 3407910256 8,16 4 1554 55.854626567 3761 Q WS 3407912304 + 480 [qemu-kvm] 8,16 4 1555 55.854630912 3761 S WS 3407912304 + 480 [qemu-kvm] 8,17 4 1556 55.863546935 3762 A WS 3407912280 + 480 <- (254,23) 33594328 8,16 4 1557 55.863547767 3762 A WS 3407914328 + 480 <- (8,17) 3407912280 8,16 4 1558 55.863548414 3762 Q WS 3407914328 + 480 [qemu-kvm] 8,16 4 1559 55.863552803 3762 S WS 3407914328 + 480 [qemu-kvm] 8,17 4 1560 55.864224593 3763 A WS 3407915312 + 480 <- (254,23) 33597360 8,16 4 1561 55.864225104 3763 A WS 3407917360 + 480 <- (8,17) 3407915312 8,16 4 1562 55.864225377 3763 Q WS 3407917360 + 480 [qemu-kvm] 8,16 4 1563 55.864228934 3763 S WS 3407917360 + 480 [qemu-kvm] 8,17 4 1564 55.866627863 3764 A WS 3407918296 + 152 <- (254,23) 33600344 8,16 4 1565 55.866628591 3764 A WS 3407920344 + 152 <- (8,17) 3407918296 8,16 4 1566 55.866628983 3764 Q WS 3407920344 + 152 [qemu-kvm] 8,16 4 1567 55.866633183 3764 S WS 3407920344 + 152 [qemu-kvm] 8,17 4 1568 55.877039182 3765 A WS 3407918456 + 480 <- (254,23) 33600504 8,16 4 1569 55.877040027 3765 A WS 3407920504 + 480 <- (8,17) 3407918456 8,16 4 1570 55.877040609 3765 Q WS 3407920504 + 480 [qemu-kvm] 8,16 4 1571 55.877044815 3765 S WS 3407920504 + 480 [qemu-kvm] 8,17 4 1572 55.877686947 3766 A WS 3407920480 + 480 <- (254,23) 33602528 8,16 4 1573 55.877687518 3766 A WS 3407922528 + 480 <- (8,17) 3407920480 8,16 4 1574 55.877687831 3766 Q WS 3407922528 + 480 [qemu-kvm] 8,16 4 1575 55.877691496 3766 S WS 3407922528 + 480 [qemu-kvm] 8,17 4 1576 55.878521409 3767 A WS 3407921616 + 480 <- (254,23) 33603664 8,16 4 1577 55.878522026 3767 A WS 3407923664 + 480 <- (8,17) 3407921616 8,16 4 1578 55.878522333 3767 Q WS 3407923664 + 480 [qemu-kvm] 8,16 4 1579 55.878526111 3767 S WS 3407923664 + 480 [qemu-kvm] 8,17 3 3957 55.886319962 487 A WS 3491537488 + 480 <- (254,25) 33333456 8,16 3 3958 55.886320831 487 A WS 3491539536 + 480 <- (8,17) 3491537488 8,16 3 3959 55.886328182 487 Q WS 3491539536 + 480 [kworker/3:1] 8,16 3 3960 55.886335232 487 S WS 3491539536 + 480 [kworker/3:1] 8,17 7 923 55.890487690 3768 A WS 3407923648 + 480 <- (254,23) 33605696 8,16 7 924 55.890488716 3768 A WS 3407925696 + 480 <- (8,17) 3407923648 8,16 7 925 55.890489350 3768 Q WS 3407925696 + 480 [qemu-kvm] 8,16 7 926 55.890494793 3768 S WS 3407925696 + 480 [qemu-kvm] 8,16 1 1 1266874889.707163736 0 C R 3390363000 + 48 [0] 8,16 1 0 1266874889.707181482 0 m N cfq3273 complete rqnoidle 0 CPU0 (8,16): Reads Queued: 1133, 20864KiB Writes Queued: 161, 912KiB Read Dispatches: 1131, 20872KiB Write Dispatches: 221, 3388KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 1551, 27852KiB Writes Completed: 481, 5332KiB Read Merges: 0, 0KiB Write Merges: 8, 32KiB Read depth: 13 Write depth: 17 IO unplugs: 1370 Timer unplugs: 0 CPU1 (8,16): Reads Queued: 1249, 22376KiB Writes Queued: 200, 1364KiB Read Dispatches: 1250, 22412KiB Write Dispatches: 290, 3080KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 1760, 31589KiB Writes Completed: 545, 5712KiB Read Merges: 0, 0KiB Write Merges: 8, 32KiB Read depth: 13 Write depth: 17 IO unplugs: 1527 Timer unplugs: 0 CPU2 (8,16): Reads Queued: 309, 4792KiB Writes Queued: 79, 456KiB Read Dispatches: 307, 4580KiB Write Dispatches: 76, 424KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 420 Timer unplugs: 0 CPU3 (8,16): Reads Queued: 354, 5708KiB Writes Queued: 160, 4056KiB Read Dispatches: 357, 5700KiB Write Dispatches: 134, 1056KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 3, 12KiB Read depth: 13 Write depth: 17 IO unplugs: 595 Timer unplugs: 0 CPU4 (8,16): Reads Queued: 82, 1652KiB Writes Queued: 136, 3040KiB Read Dispatches: 82, 1652KiB Write Dispatches: 74, 340KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 19, 76KiB Read depth: 13 Write depth: 17 IO unplugs: 243 Timer unplugs: 0 CPU5 (8,16): Reads Queued: 101, 2633KiB Writes Queued: 91, 508KiB Read Dispatches: 102, 2637KiB Write Dispatches: 86, 388KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 212 Timer unplugs: 0 CPU6 (8,16): Reads Queued: 23, 364KiB Writes Queued: 75, 7432KiB Read Dispatches: 22, 348KiB Write Dispatches: 32, 140KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 137 Timer unplugs: 0 CPU7 (8,16): Reads Queued: 38, 492KiB Writes Queued: 95, 5840KiB Read Dispatches: 38, 492KiB Write Dispatches: 42, 448KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 4, 16KiB Read depth: 13 Write depth: 17 IO unplugs: 142 Timer unplugs: 0 CPU8 (8,16): Reads Queued: 3, 148KiB Writes Queued: 66, 272KiB Read Dispatches: 3, 148KiB Write Dispatches: 33, 140KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 8, 32KiB Read depth: 13 Write depth: 17 IO unplugs: 79 Timer unplugs: 0 CPU9 (8,16): Reads Queued: 7, 92KiB Writes Queued: 52, 3612KiB Read Dispatches: 6, 88KiB Write Dispatches: 29, 120KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 64 Timer unplugs: 0 CPU10 (8,16): Reads Queued: 1, 32KiB Writes Queued: 104, 416KiB Read Dispatches: 1, 32KiB Write Dispatches: 19, 336KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 65, 260KiB Read depth: 13 Write depth: 17 IO unplugs: 60 Timer unplugs: 0 CPU11 (8,16): Reads Queued: 7, 48KiB Writes Queued: 16, 68KiB Read Dispatches: 7, 48KiB Write Dispatches: 8, 36KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 40 Timer unplugs: 0 CPU12 (8,16): Reads Queued: 1424, 23672KiB Writes Queued: 139, 2780KiB Read Dispatches: 1415, 23640KiB Write Dispatches: 180, 6788KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 1962, 31868KiB Writes Completed: 367, 8440KiB Read Merges: 0, 0KiB Write Merges: 1, 8KiB Read depth: 13 Write depth: 17 IO unplugs: 1616 Timer unplugs: 0 CPU13 (8,16): Reads Queued: 1223, 23770KiB Writes Queued: 113, 656KiB Read Dispatches: 1230, 23982KiB Write Dispatches: 141, 4020KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 1821, 37370KiB Writes Completed: 334, 5256KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 1390 Timer unplugs: 0 CPU14 (8,16): Reads Queued: 400, 6044KiB Writes Queued: 96, 2032KiB Read Dispatches: 403, 6048KiB Write Dispatches: 76, 580KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 2, 152KiB Read depth: 13 Write depth: 17 IO unplugs: 517 Timer unplugs: 0 CPU15 (8,16): Reads Queued: 404, 9400KiB Writes Queued: 96, 2788KiB Read Dispatches: 405, 9420KiB Write Dispatches: 82, 636KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 513 Timer unplugs: 0 CPU16 (8,16): Reads Queued: 96, 1568KiB Writes Queued: 59, 316KiB Read Dispatches: 96, 1568KiB Write Dispatches: 56, 1288KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 170 Timer unplugs: 0 CPU17 (8,16): Reads Queued: 135, 2908KiB Writes Queued: 50, 212KiB Read Dispatches: 135, 2908KiB Write Dispatches: 39, 168KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 202 Timer unplugs: 0 CPU18 (8,16): Reads Queued: 28, 248KiB Writes Queued: 51, 1784KiB Read Dispatches: 29, 252KiB Write Dispatches: 39, 200KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 86 Timer unplugs: 0 CPU19 (8,16): Reads Queued: 43, 804KiB Writes Queued: 97, 9788KiB Read Dispatches: 41, 788KiB Write Dispatches: 32, 916KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 138 Timer unplugs: 0 CPU20 (8,16): Reads Queued: 10, 192KiB Writes Queued: 29, 2128KiB Read Dispatches: 10, 192KiB Write Dispatches: 9, 44KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 39 Timer unplugs: 0 CPU21 (8,16): Reads Queued: 12, 300KiB Writes Queued: 35, 2168KiB Read Dispatches: 12, 300KiB Write Dispatches: 21, 124KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 50 Timer unplugs: 0 CPU22 (8,16): Reads Queued: 3, 16KiB Writes Queued: 7, 440KiB Read Dispatches: 3, 16KiB Write Dispatches: 2, 44KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 14 Timer unplugs: 0 CPU23 (8,16): Reads Queued: 1, 4KiB Writes Queued: 7, 40KiB Read Dispatches: 1, 4KiB Write Dispatches: 6, 36KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 0, 0KiB Writes Completed: 0, 0KiB Read Merges: 0, 0KiB Write Merges: 0, 0KiB Read depth: 13 Write depth: 17 IO unplugs: 15 Timer unplugs: 0 Total (8,16): Reads Queued: 7086, 128127KiB Writes Queued: 2014, 53108KiB Read Dispatches: 7086, 128127KiB Write Dispatches: 1727, 24740KiB Reads Requeued: 0 Writes Requeued: 0 Reads Completed: 7094, 128679KiB Writes Completed: 1727, 24740KiB Read Merges: 0, 0KiB Write Merges: 118, 620KiB IO unplugs: 9639 Timer unplugs: 0 Throughput (R/W): 0KiB/s / 0KiB/s Events (8,16): 193565 entries Skips: 21833 forward (8033027806 - 100.0%)
-- libvir-list mailing list libvir-list@xxxxxxxxxx https://www.redhat.com/mailman/listinfo/libvir-list