Re: Shutting down a VM with Kernel 4.14 will sometime hang and a reboot is the only way to recover.

[Dave Hill <dhill@xxxxxxxxxx>]

On 11/28/2017 8:52 PM, Jason Wang wrote:
> On 2017年11月29日 02:00, David Hill wrote:
> > On 2017-11-27 02:38 PM, David Hill wrote:
> > > On 2017-11-26 10:44 PM, Jason Wang wrote:
> > > > On 2017年11月25日 00:22, David Hill wrote:
> > > > > The VMs all have 2 vNICs ... and this is the hypervisor:
> > > > >
> > > > > [root@zappa ~]# brctl show
> > > > > bridge name    bridge id        STP enabled    interfaces
> > > > > virbr0        8000.525400914858    yes        virbr0-nic
> > > > >                             vnet0
> > > > >                             vnet1
> > > > >
> > > > >
> > > > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state 
> > > > > UNKNOWN group default qlen 1000
> > > > >     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
> > > > >     inet 127.0.0.1/8 scope host lo
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet6 ::1/128 scope host
> > > > >        valid_lft forever preferred_lft forever
> > > > > 2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
> > > > > UP group default qlen 1000
> > > > >     link/ether 84:2b:2b:13:f2:91 brd ff:ff:ff:ff:ff:ff
> > > > >     inet redacted/24 brd 173.178.138.255 scope global dynamic eno1
> > > > >        valid_lft 48749sec preferred_lft 48749sec
> > > > >     inet6 fe80::862b:2bff:fe13:f291/64 scope link
> > > > >        valid_lft forever preferred_lft forever
> > > > > 3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state 
> > > > > UP group default qlen 1000
> > > > >     link/ether 84:2b:2b:13:f2:92 brd ff:ff:ff:ff:ff:ff
> > > > >     inet 192.168.1.3/24 brd 192.168.1.255 scope global eno2
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet6 fe80::862b:2bff:fe13:f292/64 scope link
> > > > >        valid_lft forever preferred_lft forever
> > > > > 4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
> > > > > noqueue state UP group default qlen 1000
> > > > >     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
> > > > >     inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
> > > > >
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.10/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.11/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.12/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.15/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.16/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.17/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.18/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.31/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.32/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.33/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.34/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.35/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.36/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.37/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.45/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.46/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.47/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.48/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.49/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.50/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet 192.168.122.51/32 scope global virbr0
> > > > >        valid_lft forever preferred_lft forever
> > > > > 5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel 
> > > > > master virbr0 state DOWN group default qlen 1000
> > > > >     link/ether 52:54:00:91:48:58 brd ff:ff:ff:ff:ff:ff
> > > > > 125: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1360 
> > > > > qdisc fq_codel state UNKNOWN group default qlen 100
> > > > >     link/none
> > > > >     inet 10.10.122.28/21 brd 10.10.127.255 scope global tun0
> > > > >        valid_lft forever preferred_lft forever
> > > > >     inet6 fe80::1f9b:bfd4:e9c9:2059/64 scope link stable-privacy
> > > > >        valid_lft forever preferred_lft forever
> > > > > 402: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
> > > > > fq_codel master virbr0 state UNKNOWN group default qlen 1000
> > > > >     link/ether fe:54:00:09:27:39 brd ff:ff:ff:ff:ff:ff
> > > > >     inet6 fe80::fc54:ff:fe09:2739/64 scope link
> > > > >        valid_lft forever preferred_lft forever
> > > > > 403: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc 
> > > > > fq_codel master virbr0 state UNKNOWN group default qlen 1000
> > > > >     link/ether fe:54:00:ea:6b:18 brd ff:ff:ff:ff:ff:ff
> > > > >     inet6 fe80::fc54:ff:feea:6b18/64 scope link
> > > > >        valid_lft forever preferred_lft forever
> > > >
> > > > I could not reproduce this locally by simply running netperf 
> > > > through a mlx4 card. Some more questions:
> > > >
> > > > - What kind of workloads did you run in guest?
> > > > - Did you meet this issue in a specific type of network card (I 
> > > > guess broadcom is used in this case)?
> > > > - Virbr0 looks like a bridge created by libvirt that did NAT and 
> > > > other stuffs, can you still hit this issue if you don't use virbr0?
> > > >
> > > > And what's more important, zerocopy is known to have issues, for 
> > > > production environment, need to disable it through vhost_net module 
> > > > parameters.
> > > >
> > > > Thanks
> > >
> > > I'm deploying an overcloud through a undercloud virtual machine... 
> > > The VM has 4vCPUs and 16GB of RAM as well as to virtio nics so I'm 
> > > using only virtual hardware here.
> > > I spawn 7 VMs on the hypervisor and deploy an overcloud using 
> > > tripleo on them ... everything's virtual and if I remove the bridge, 
> > > then I'll have to configure each VMs differently.
> > > The load is quite high on the VM that won't shutdown but when I shut 
> > > it down, it's doing nothing ...   This is a hard bug to troubleshoot 
> > > and I can't bisect the kernel because at some
> > > point the system simply won't boot properly.
> >
> > I've disabled zerocopy with the following:
> >
> > [root@zappa modprobe.d]# cat vhost-net.conf
> > options vhost_net  experimental_zcopytx=0
> >
> >
> > And I haven't reproduce this issue so far.   The problem I have right 
> > now is that experimental_zcopytx has been enabled by default with 
> > this commit:
> >
> > commit f9611c43ab0ddaf547b395c90fb842f55959334c
> > Author: Michael S. Tsirkin <mst@xxxxxxxxxx>
> > Date:   Thu Dec 6 14:56:00 2012 +0200
> >
> >     vhost-net: enable zerocopy tx by default
> >
> >     Zero copy TX has been around for a while now.
> >     We seem to be down to eliminating theoretical bugs
> >     and performance tuning at this point:
> >     it's probably time to enable it by default so that
> >     most users get the benefit.
> >
> >     Keep the flag around meanwhile so users can experiment
> >     with disabling this if they experience regressions.
> >     I expect that we will remove it in the future.
> >
> >     Signed-off-by: Michael S. Tsirkin <mst@xxxxxxxxxx>
> >
> > I'll try some more pass in producing this issue and I'll keep you 
> > posted.
> >
> > Thank you very much,
> >
> > David Hill
>
> Thanks. Zerocopy is disabled by several distribution by default. For 
> upstream, the only reason to let it on is to hope more developers can 
> help and fix the issues.
So I never hit this issue with previous kernel and this issue started 
happening with the v4.14-rc series.   I'm using rawhide so perhaps this 
is why it isn't disabled by default but I have to mention it's an update 
of FC25 up to FC28 and it never got disabled.
Perhaps it should be disabled in Fedora too if it's not the case... I'm 
not sure this is the place to discuss this ... is it?