The issue about adding multipath device's targets into qemu-pr-helper's namespace

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi all,

I have a namespace question about passthrough disk(multipath device).
In case of enabling namespace and cgroups in qemu.conf, The target(s) of the multipath device won't be added into qemu-pr-helper's namespace under certain
situation, It causes the PERSISTENT RESERVE command failure in guest.

While user starts a vm,
To build namespace, The qemuDomainSetupDisk() will be invoked via threadA(this
thread id will be the qemu's pid),
To build cgroup, The qemuSetupImagePathCgroup() will be invoked via threadB.

Both of the functions invoke the virDevMapperGetTargets() trying to parse a
multipath device to target paths string, Then fill the targetPaths[].

The issue I experienced is:
After libvirtd started, Everything works well for the first booted vm which has
the passthrough multipath device.
But If I shut it down & start it again, OR keep it running & start another vm which has other passthrough multipath device, Then the target(s) of the fresh started vm won't be added into the related qemu-pr-helper's namespace and it
causes PERSISTENT RESERVE command failure in the corresponding guest.
I digged into code, In this situation, The targetPaths[] in qemuDomainSetupDisk()
won't be filled, it keeps NULL after virDevMapperGetTargets() returns.
The virDevMapperGetTargets doesn't fill targetPaths[] because the dm_task_run()
of libdevmapper returns 0 with errno 9(Bad file descriptor).
So far, I don't understand why the dm_task_run() return 0 in this situation. BTW, The virDevMapperGetTargets() can always successfully fill the targetPaths[]
in qemuSetupImagePathCgroup().

Please refer to the following 2 tests:
The multipath configuration on host:
host:~ # multipath -l
vm1-data (3600140582d9024bc13f4b8db5ff12de0) dm-11 FreeNAS,lv68
size=6.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 2:0:0:2 sdd 8:48 active undef running
vm2-data (36001405fc5f29ace3ec4fb8acd32aae5) dm-8 FreeNAS,lv46
size=4.0G features='0' hwhandler='1 alua' wp=rw
`-+- policy='service-time 0' prio=0 status=active
  `- 2:0:0:1 sde 8:64 active undef running

===================================================================
Test A:
host:~ # systemctl restart libvirtd
host:~ # virsh list
 Id   Name   State
--------------------

host:~ #
host:~ # virsh domblklist vm1
 Target   Source
------------------------------------------
 vda      /opt/vms/vm1/disk0.qcow2
 sda      /dev/mapper/vm1-data

host:~ #
host:~ # virsh start vm1
Domain vm1 started

host:~ # virsh list
 Id   Name        State
---------------------------
 1    vm1        running

host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 16:30 /dev/sdd
host:~ # exit
exit
host:~ #

vm1:~ # lsscsi
[0:0:0:0]    disk    FreeNAS  lv68             0123   /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
  FreeNAS   lv68              0123
  Peripheral device type: disk
  PR generation=0x0, there are NO registered reservation keys
vm1:~ #

host:~ # virsh shutdown vm1
Domain vm1 is being shutdown

host:~ # virsh list
 Id   Name   State
--------------------

host:~ #
host:~ # virsh start vm1
Domain vm1 started

host:~ # virsh list
 Id   Name        State
---------------------------
 2    vm1        running

host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #

vm1:~ # sg_persist --in -k /dev/sda
  FreeNAS   lv68              0123
  Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm1:~ #
===================================================================
Test B:
host:~ # systemctl restart libvirtd
host:~ # virsh list
 Id   Name   State
--------------------

host:~ #
host:~ # virsh domblklist vm1
 Target   Source
------------------------------------------
 vda      /opt/vms/vm1/disk0.qcow2
 sda      /dev/mapper/vm1-data

host:~ #
host:~ # virsh start vm1
Domain vm1 started

host:~ # virsh list
 Id   Name        State
---------------------------
 1    vm1        running

host:~ # nsenter -t $(pidof qemu-pr-helper) -a bash
host:~ # ls -l /dev/sd*
brw-rw---- 1 root disk 8, 48 Jul 14 17:28 /dev/sdd
host:~ # exit
exit
host:~ #

vm1:~ # lsscsi
[2:0:0:0]    disk    FreeNAS  lv68             0123   /dev/sda
vm1:~ #
vm1:~ # sg_persist --in -k /dev/sda
  FreeNAS   lv68              0123
  Peripheral device type: disk
  PR generation=0x0, there are NO registered reservation keys
vm1:~ #

host:~ # virsh list
 Id   Name        State
---------------------------
 1    vm1        running

host:~ #
host:~ # virsh domblklist vm2
 Target   Source
------------------------------------------
 vda      /opt/vms/vm2/disk0.qcow2
 sda      /dev/mapper/vm2-data

host:~ #
host:~ # virsh start vm2
Domain vm2 started

host:~ # virsh list
 Id   Name        State
---------------------------
 1    vm1        running
 2    vm2        running

host:~ # nsenter -t $(qemu-pr-helper pid of vm2) -a bash
host:~ # ls -l /dev/sd*
ls: cannot access '/dev/sd*': No such file or directory
host:~ # exit
exit
host:~ #

vm2:~ # lsscsi
[0:0:0:0]    disk    FreeNAS  lv46             0123   /dev/sda
vm2:~ #
vm2:~ # sg_persist --in -k /dev/sda
  FreeNAS   lv46              0123
  Peripheral device type: disk
PR in (Read keys): Aborted command
Aborted command
vm2:~ #
===================================================================

Any comments will be much appreciated.

Thanks in advance,
Lin




[Index of Archives]     [Virt Tools]     [Libvirt Users]     [Lib OS Info]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [KDE Users]     [Fedora Tools]

  Powered by Linux