Hi all,
I hope i'm in the right place to ask for help, but here goes.
I'm trying to setup a virtualized network video recorder (NVR) for a video
surveillance system and am encountering what seems to be network related
issues.
In the NVR software i see that many video frames are lost and the
workstation (which projects the video streams on a videowall) has stuttering
video.
It is a KVM based setup with a Fedora 16 guest running the NVR software.
There are around 90 camera's sending video to the NVR, the camera's are
already limited to 2mbit streams. This is all UDP traffic.
As a host i'm using Debian Linux 6.0 squeeze with the latest 2.6.32-5-amd64
kernel and qemu-kvm (1.1.2+dfsg-5~bpo60+1) from the backports repo.
The host has a intel pro/1000 PT Quad Port LP Server Adapter (using the
e1000e driver) and i created two bridges (br0 and br1)
The gigabit ports should all be capable of handling 90x2mbit=180mbit UDP
traffic
br0 uses eth0 and is used only for incoming UDP traffic (from the camera's)
br1 uses eth1 and is used to send the live video streams to the workstation
(videowall), this is TCP traffic.
The guest has two virtio-nic's, one bridged to br0 and one bridged to br1.
Now what i'm seeing is LOADS of overruns on the tap0 interface (this is the
interface bridged to br0, incoming UDP traffic):
tap0 Link encap:Ethernet HWaddr 5a:2a:98:6d:38:bb
inet6 addr: fe80::582a:98ff:fe6d:38bb/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:187935991 errors:0 dropped:0 overruns:0 frame:0
TX packets:54557072995 errors:0 dropped:0 overruns:211649232
carrier:0
collisions:0 txqueuelen:500
RX bytes:36940172592 (34.4 GiB) TX bytes:72782294943352 (66.1
TiB)
Searching google/maillinglists i tried increasing the txqueuelen value to
higher values (2500/10000) for the tap0 interface, this helps a little but
doesn't eliminate the problem.
I searched the kvm mailinglist archives about these overruns and I found a
thread which links to a bugreport from 2009 which is still open and seems to
be related:
http://sourceforge.net/p/kvm/bugs/383/
I don't have a network that stops working but i'm loosing UDP packets and it
is happening in bursts where the network seems to freeze for a bit and this
results in the overruns counter on tap0 increasing a lot.
I was able to reproduce this behaviour on a test rig with rather similar
specs by using iperf.
On the fedora 16 guest i ran: iperf -s -u
on a remote client on the same network in ran: iperf -c <kvm guest ip> -i 1
-u -P 90 -b 5m -t 500
-c = client
-u = udp
-i 1 = 1 second interval
-P 90 is 90 paralel streams (90 simulating camera's)
-b 5m = 5mbit per stream, which should be the value the camera's are
supposed to be running at.
-t 500 = run this test for 500 seconds.
I noticed overruns started occuring at random moments and happening in
bursts like with the live system.
I tried running the same tests with the hostsystem's kernel upgraded to the
backports kernel:
linux-image-3.2.0-0.bpo.4-amd64 (3.2.35-2~bpo60+1)
which resulted in pretty much the same numbers/behaviour.
I went a little further and found out something rather peculiar:
using the experimental kernel:
linux-image-3.8-trunk-amd64 (3.8-1~experimental.1)
I repeated the same test and there WASN'T ANY overrun encountered.
I started increasing the mbit/s for the streams and at 8/9mbit/s i started
getting DROPPED frames at the tap0 interface but still not a single overrun.
Unfortunatly, I wasn't able to test the 3.8 kernel on the live system yet,
to see if that solves the problem.
Is this still a known issue or was this behaviour fixed in the 3.8 kernel?
As i'm not really fond of running an experimental kernel, are there any
commits which are related to this issue between 3.2 and 3.8 kernel which I
can use on the 3.2 kernel?
If there is need for more information/testing, i'm happy to provide.
I hope someone can help me troubleshoot this.
Thanks in advance,
Caspar Smit
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
