I have a strange slowness which affects some guests after they are
running for some time. "Slowness" can happen a few hours after guest
start, or, a couple of days after guest start.
What do I mean by "slowness"?
This is how long it takes to log in via SSH to an unaffected guest -
below a second:
$ time ssh backupuser@normal_guest exit
0.02user 0.01system 0:00.67elapsed 4%CPU (0avgtext+0avgdata 0maxresident)
Now, let's try to log in to the affected guest running on the same host
- more than 12 seconds:
$ time ssh backupuser@slow_guest exit
0.02user 0.01system 0:12.56elapsed 0%CPU (0avgtext+0avgdata 0maxresident)
If I log in via SSH to the affected guest, any key presses lag a second
or two.
This is actually weird - if I run something IO intensive on the guest,
the login is much faster (running CPU-intensive tasks makes no difference):
guest# dd if=/dev/vda of=/dev/null
$ time ssh backupuser@slow_guest exit
0.02user 0.00system 0:00.70elapsed 2%CPU (0avgtext+0avgdata 0maxresident)
Also, running "ping -f <slow_guest>" helps a lot and SSH logins are fast.
Look at the difference here - 7470ms vs 139183ms (and packet losses):
# ping -f -c 10000 normal_guest
10000 packets transmitted, 10000 received, 0% packet loss, time 7470ms
rtt min/avg/max/mdev = 0.443/0.709/6.487/0.112 ms, ipg/ewma 0.747/0.716 ms
# ping -f -c 10000 slow_guest
10000 packets transmitted, 9934 received, 0% packet loss, time 139183ms
rtt min/avg/max/mdev = 0.470/14.337/50.455/5.409 ms, pipe 4, ipg/ewma
13.919/14.788 ms
CPU-intensive tasks are as fast as on unaffected guests.
Reading from /dev/vda is as fast as on unaffected guests.
So the only thing broken seems to be the network.
Rebooting the guest does not help - it is still slow.
The only thing that helps is stopping the guest and starting it again
(i.e., stopping kvm process and starting a new one).
Is there an explanation to this phenomenon? Looks like a problem with
virtio drivers somewhere, or?
The host is running kvm-83.
Affected guests are running 2.6.27.14 kernels and use virtio drivers.
The problem happens only _sometimes_. Out of 9 guests I have running on
this host, I saw this problem only on 3 guests. I never saw this
happening on more than one guest at a time.
All three have 512 MB memory assigned, other guests have less memory.
--
Tomasz Chmielewski
http://wpkg.org
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
