Karel, Thanks for the detailed report. Would you mind re-posting this to the xen-devel mailing list? Thanks, -Georeg On Thu, May 24, 2018 at 9:47 AM, Karel Hendrych <k+centosvirt@xxxxxxxxx> wrote: > Bump. Folks, any ideas? > > Cheers > Karel > > > On 22.5.2018 11:33, Karel Hendrych wrote: >> >> Hi, I am seeing frequent libvirtd hangs (clients not responding) after >> last CentOS6-Xen update : >> >> libvirt-libs-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-network-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-nwfilter-4.1.0-2.xen46.el6.x86_64 >> libgcc-4.4.7-18.el6_9.2.x86_64 >> 2:qemu-img-0.12.1.2-2.503.el6_9.5.x86_64 >> libvirt-daemon-driver-storage-core-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-secret-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-interface-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-nodedev-4.1.0-2.xen46.el6.x86_64 >> 10:centos-release-xen-common-8-4.el6.x86_64 >> xen-licenses-4.6.6-12.el6.x86_64 >> xen-libs-4.6.6-12.el6.x86_64 >> libvirt-daemon-driver-libxl-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-xen-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-qemu-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-storage-gluster-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-storage-logical-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-storage-mpath-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-storage-disk-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-storage-scsi-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-storage-iscsi-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-storage-4.1.0-2.xen46.el6.x86_64 >> libstdc++-4.4.7-18.el6_9.2.x86_64 >> libvirt-daemon-config-nwfilter-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-config-network-4.1.0-2.xen46.el6.x86_64 >> libvirt-daemon-driver-lxc-4.1.0-2.xen46.el6.x86_64 >> libvirt-client-4.1.0-2.xen46.el6.x86_64 >> linux-firmware-20171215-82.git2451bb22.el6.noarch >> 12:dhcp-common-4.1.1-53.P1.el6.centos.4.x86_64 >> 12:dhclient-4.1.1-53.P1.el6.centos.4.x86_64 >> libvirt-4.1.0-2.xen46.el6.x86_64 >> 10:centos-release-xen-46-8-4.el6.x86_64 >> 10:centos-release-xen-44-8-4.el6.x86_64 >> tzdata-2018e-3.el6.noarch >> libgomp-4.4.7-18.el6_9.2.x86_64 >> kernel-4.9.86-30.el6.x86_64 >> xen-hypervisor-4.6.6-12.el6.x86_64 >> xen-runtime-4.6.6-12.el6.x86_64 >> xen-4.6.6-12.el6.x86_64 >> libvirt-daemon-xen-4.1.0-2.xen46.el6.x86_64 >> >> Remedy is to kill -9 libvirtd and start again. Issue can be replicated >> within few domU starts. Usually libvirtd hangs when domU is bringing up xen >> drivers or something around udev, like: >> >> xen_netfront: Initialising Xen virtual ethernet driver >> >> I've been looking into libvirtd strace and debug logs, so far most >> suspicious in libvirtd debug log is this: >> >> libvirtd.log:2018-05-22 08:32:44.760+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-7' >> libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-6' >> libvirtd.log:2018-05-22 08:32:44.761+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-4' >> libvirtd.log:2018-05-22 08:32:44.762+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-5' >> libvirtd.log:2018-05-22 08:32:44.763+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-2' >> libvirtd.log:2018-05-22 08:32:44.764+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/tx-3' >> libvirtd.log:2018-05-22 08:32:44.765+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-6' >> libvirtd.log:2018-05-22 08:32:44.766+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-5' >> libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-4' >> libvirtd.log:2018-05-22 08:32:44.767+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-7' >> libvirtd.log:2018-05-22 08:32:44.768+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-2' >> libvirtd.log:2018-05-22 08:32:44.769+0000: 25455: debug : >> udevRemoveOneDevice:1289 : Failed to find device to remove that has udev >> name '/sys/devices/vif-24-0/net/vif24.0/queues/rx-3' >> >> I could not get rid of this by reducing amount of driver queues (not sure >> if that applies to PV) >> >> Is someone out there seeing similar issues? Anyone perhaps interested in >> reviewing full debug log / strace ? >> >> Cheers >> Karel >> >> _______________________________________________ >> CentOS-virt mailing list >> CentOS-virt@xxxxxxxxxx >> https://lists.centos.org/mailman/listinfo/centos-virt > > _______________________________________________ > CentOS-virt mailing list > CentOS-virt@xxxxxxxxxx > https://lists.centos.org/mailman/listinfo/centos-virt _______________________________________________ CentOS-virt mailing list CentOS-virt@xxxxxxxxxx https://lists.centos.org/mailman/listinfo/centos-virt
