Secondary disk unattached if guest unresponsive and killed with 'virsh destroy'

[Mathew Crane <mathew.crane@xxxxxxxxx>]

I have a very strange issue that has plagued our KVM-based hypervisor infrastructure. 

Environment:
- All EL7-based libvirt releases since EL7.0
- Fully patched EL7-based KVM guest on a fully patched EL7-based KVM host.

- Secondary disk attached via 'virsh attach-device guest /vm/guest-with-secondary-disk-01/disk2.xml --persistent' (note --persistent flag here)

Problem Description:

- Guest becomes unresponsive/hangs due to issue on guest. Typically the scenario is that PID1 hangs or goes into a bad state due to a misbehaving application.
- Before virsh destroy, 'virsh dumpxml guest | grep vd[ab]' returns the expected filesystems.

- Guest is destroyed from the hypervisor via 'virsh destroy guest'. 
- With guest still powered off, 'virsh dumpxml guest | grep vd[ab]' only shows 'vda'. 'vdb' now unattached.
- To fix, requires re-run of 'virsh attach-device guest guest-with-secondary-disk-01/disk2.xml --persistent'

Expected Behavior:
- 'virsh destroy guest' does not drop the secondary disk attached to the guest if the node is in a hung state.

Other notes:
- If guest is not in a hung state, 'virsh destroy' does not drop the secondary disk.
- 'virsh reset guest' does not exhibit the same behavior when guest is hung - vdb remains attached.
- No logs produced in /var/log/libvirt indicating an issue, even with debug logging enabled.

Is there any obvious I'm missing? Is this expected behavior?

Guest XML via 'virsh dumpxml guest':

<domain type='kvm' id='17'>

  <name>guest-with-secondary-disk-01</name>

  <uuid>ab578b51-3afd-4d01-86ab-a16b1b953919</uuid>

  <memory unit='KiB'>4194304</memory>

  <currentMemory unit='KiB'>4194304</currentMemory>

  <vcpu placement='static'>1</vcpu>

  <resource>

    <partition>/machine</partition>

  </resource>

  <os>

    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>

    <boot dev='hd'/>

    <bootmenu enable='yes'/>

  </os>

  <features>

    <acpi/>

    <apic/>

    <pae/>

  </features>

  <clock offset='utc'/>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/libexec/qemu-kvm</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2' cache='none'/>

      <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01.img'/>

      <backingStore/>

      <target dev='vda' bus='virtio'/>

      <alias name='virtio-disk0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </disk>

    <disk type='file' device='disk'>

      <driver name='qemu' type='qcow2' cache='none'/>

      <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/>

      <backingStore/>

      <target dev='vdb' bus='virtio'/>

      <alias name='virtio-disk1'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>

    </disk>

    <controller type='usb' index='0' model='piix3-uhci'>

      <alias name='usb'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'>

      <alias name='pci.0'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <alias name='virtio-serial0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='01:02:03:99:99:99'/>

      <source bridge='bridge_99'/>

      <target dev='vnet0'/>

      <model type='virtio'/>

      <alias name='eth0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

    </interface>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/guest-with-secondary-disk-01.agent'/>

      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>

      <alias name='channel0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <input type='mouse' bus='ps2'>

      <alias name='input0'/>

    </input>

    <input type='keyboard' bus='ps2'>

      <alias name='input1'/>

    </input>

    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1' keymap='en-us'>

      <listen type='address' address='127.0.0.1'/>

    </graphics>

    <video>

      <model type='cirrus' vram='16384' heads='1' primary='yes'/>

      <alias name='video0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </video>

    <memballoon model='virtio'>

      <alias name='balloon0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </memballoon>

    <rng model='virtio'>

      <backend model='random'>/dev/random</backend>

      <alias name='rng0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>

    </rng>

  </devices>

  <seclabel type='dynamic' model='selinux' relabel='yes'>

    <label>system_u:system_r:svirt_t:s0:c259,c495</label>

    <imagelabel>system_u:object_r:svirt_image_t:s0:c259,c495</imagelabel>

  </seclabel>

  <seclabel type='dynamic' model='dac' relabel='yes'>

    <label>+107:+107</label>

    <imagelabel>+107:+107</imagelabel>

  </seclabel>

</domain>

Secondary disk XML: 

<disk type='file' device='disk'>

  <driver name='qemu' type='qcow2' cache='none'/>

  <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/>

  <target dev='vdb'/>

</disk>

_______________________________________________
libvirt-users mailing list
libvirt-users@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/libvirt-users