On Sat, Aug 18, 2012 at 01:25:05AM +0200, Bernd Schubert wrote: > On 08/12/2012 01:45 PM, Michael S. Tsirkin wrote: > > On Mon, Jul 30, 2012 at 08:08:31PM +0200, Bernd Schubert wrote: > >> On 07/30/2012 07:33 PM, Bernd Schubert wrote: > >>> Hello Stefan, > >>> > >>> Stefan Hajnoczi <stefanha <at> gmail.com> writes: > >>>> > >>>> On Wed, Jan 11, 2012 at 4:18 PM, Bernd Schubert > >>>> <bernd.schubert <at> itwm.fraunhofer.de> wrote: > >>>>> On 01/11/2012 05:04 PM, Stefan Hajnoczi wrote: > >>>>>> Try pinging the host's IP address from inside the guest. Run tcpdump > >>>>>> on the guest's tap interface from the host and observe whether or not > >>>>>> you see any packets being sent from the guest. > >>>>> > >>> > >>> > >>> sorry for my terribly late reply. As usual I got distracted by too many other > >>> things and then returned the hardware I was running the VMs on. My new desktop > >>> system is better suitable to run kvm and I can easily reproduce it now with 3.5 > >>> on host and guest side. So its not fixed in recent versions yet. > >>> > >>> > >>>>> > >>>>> Seems arp requests are still going out, but then don't go in: > >>>>> > >>>>> 17:16:21.202547 ARP, Reply 192.168.123.1 is-at 00:25:90:38:09:cd (oui > >>>>> Unknown), length 28 > >>>>> 17:16:21.538724 ARP, Request who-has squeeze1 tell squeeze3, length 28 > >>>>> 17:16:21.539026 ARP, Reply squeeze1 is-at 52:54:00:12:34:11 (oui Unknown), > >>>>> length 28 > >>>>> 17:16:22.200912 ARP, Request who-has 192.168.123.1 tell squeeze3, length 28 > >>>> > >>>> Okay, so it seems networking from the tap device and beyond is fine. > >>>> > >>>>>> rmmod virtio_net inside the guest and then modprobe virtio_net again. > >>>>>> See if network connectivity is restored (remember to rerun DHCP or > >>>>>> whatever, if necessary). > >>>>> > >>>>> > >>>>> Yep, that makes it work again. But probably is not the real solution ;) > >>>> > >>>> It's just another piece of information which helps debug this :). At > >>>> least nothing has wedged itself into an unrecoverable state. > >>>> > >>>> When you said the problem happens without vhost, did you explicitly > >>>> run vhost=off? Or did you just omit "vhost=on"? > >>> > >>> It was definitely off and I can confirm that it also locks up with vhost=on and > >>> vhost=off with 3.5. > >>> > >>>> > >>>> This sounds like a guest kernel/driver issue. I recommend testing > >>>> git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git in > >>>> the guest to see if this has already been fixed. > >>>> > >>>> If you have the -dbg RPMs installed it may be possible to insert a > >>>> probe into the virtio_net kernel module and observe receive > >>>> interrupts. This does require the right kernel CONFIG_ but you might > >>>> already have it enabled: > >>>> > >>>> $ sudo perf probe --add skb_recv_done > >>>> $ sudo perf record -e probe:skb_recv_done -a > >>>> ...send some packets to the guest... > >>>> ^C > >>>> $ sudo perf script > >>>> > >>>> If you see no skb_recv_done events then the guest driver is not > >>>> receiving a notification when packets are received. > >>>> > >>>> You can find more about how to use perf-probe(1) at > >>>> http://blog.vmsplice.net/2011/03/how-to-use-perf-probe.html. > >>> > >>> Ah nice, I would have used systemtap, but always wanted to check how to do it > >>> with perf :) > >>> > >>> So once the virtio NIC has locked up, I don't get any events from it anymore - > >>> until I remove/re-insert the virtio module (including ifup/ifdown). I will try > >>> to find some time later on this week to look into it again. > >>> Any further ideas how to proceed (I haven't even checked yet how virtio works at > >>> all...). > >> > >> > >> I took a quick glance where skb_recv_done is registered at all and > >> traced it back to vp_find_vqs(). Looking into that function I > >> noticed MSI and so tried to boot with pci=nomsi. And indeed I > >> guessed it right, with pci=nomsi I don't get any lockups anymore. > >> Am I the only one booting kvm-qemu usually with enabled MSI? > >> > >> Cheers, > >> Bernd > > > > No :) > > > > I am guessing it has to do with OOM handling in the guest - > > it is tested very little but maybe your guest is such that atomic > > pool gets exhausted for some reason. > > Could you pls check whether refill_work runs by tracing it? > > This is our OOM handler. > > > > > > Just checked it, it does not show up in perf script output. > > > Cheers, > Bernd When running with vhost-net on, if you enable DEBUG in *host* kernel build (or set CONFIG_DYNAMIC_DEBUG and enable messages for the vhost_net module) pr_debug will output some debug messages if guest bug is detected. Can you try this? -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
