On 11-07-28 04:33 PM, James Bottomley wrote:
On Thu, 2011-07-28 at 10:05 -0700, Roland Dreier wrote:
Hi,
I'm seeing an issue with the current design of our enclosure
handling. In a system with a bunch of drives in an enclosure, it's
definitely helpful to have a way to go from sdXX to which slot in the
enclosure that drive is in, and that's what the symlink
/sys/block/sdXX/device/enclosure_device:NN provides.
However, in a system with multiple paths to the enclosure, eg an HBA
with two external SAS ports, both connected to a SAS expander in a
JBOD, ie in lame ASCII graphics, something like:
+-----+ /-- drv1
| | +-----+ /--- .
| |==SAS==| |-/---- .
| HBA | | exp |------ .
| |==SAS==| |-\---- .
| | +-----+ \--- .
+-----+ \-- drvN
So this configuration should form a single wide port and thus not
actually be multiple paths. However, if you have two HBAs instead of
one (or a non-SAS HBA), I grant it becomes multipath.
Here are some conflicting results for the single HBA case.
I tested two LSI HBAs separately, each with 8 phys and wired 5
of those phys to a LSI SAS-2 expander.
Here are the results for a SAS-1.1 (3 Gbps) 3444E HBA seen
from a SMP DISCOVER on the expander:
phy 5:T:attached:[5001517e85c3efe5:00 t(SATA)] 3 Gbps
phy 7:T:attached:[5000c500215725bd:00 t(SSP)] 6 Gbps
phy 12:T:attached:[500605b00006f263:03 i(SSP+STP+SMP)] 3 Gbps
phy 20:T:attached:[500605b00006f264:04 i(SSP+STP+SMP)] 3 Gbps
phy 21:T:attached:[500605b00006f264:05 i(SSP+STP+SMP)] 3 Gbps
phy 22:T:attached:[500605b00006f264:06 i(SSP+STP+SMP)] 3 Gbps
phy 23:T:attached:[500605b00006f264:07 i(SSP+STP+SMP)] 3 Gbps
phy 24:D:attached:[5001517e85c3effd:00 V i(SSP+SMP) t(SSP)] 6 Gbps
That is two ports:
- a narrow port [HBA phy 3 to expander phy 12]
note the different SAS address of HBA phy 3: 500605b00006f263
- a wide port [HBA phys 4-7 to expander phys 20-23]
Should the HBA report as two hosts?? [It only reports as one host
in my test.]
So if a SAS HBA can't handle a wide port with more than
4 phys, it can just change the SAS addresses on one or more
phys at the HBA end.
Here are the results for a SAS-2 (6 Gbps) 9212 HBA seen
from a SMP DISCOVER on the same expander:
phy 5:T:attached:[5001517e85c3efe5:00 t(SATA)] 3 Gbps
phy 7:T:attached:[5000c500215725bd:00 t(SSP)] 6 Gbps
phy 12:T:attached:[500605b001d0d3e0:04 i(SSP+STP+SMP)] 6 Gbps
phy 20:T:attached:[500605b001d0d3e0:03 i(SSP+STP+SMP)] 6 Gbps
phy 21:T:attached:[500605b001d0d3e0:02 i(SSP+STP+SMP)] 6 Gbps
phy 22:T:attached:[500605b001d0d3e0:01 i(SSP+STP+SMP)] 6 Gbps
phy 23:T:attached:[500605b001d0d3e0:00 i(SSP+STP+SMP)] 6 Gbps
phy 24:D:attached:[5001517e85c3effd:00 V i(SSP+SMP) t(SSP)] 6 Gbps
Now that is a single wide port (5 phys wide) since all
5 expander phys have the same SAS address (not shown)
and all 5 HBA phys have the same address.
we have two paths to each drive, so each gets two names, sdXX and
sdYY. However, in drivers/misc/enclosure.c, the code only allows one
device in each component and so what happens is that sdXX gets
discovered, then gets an enclosure_device:NN link, then sdYY is
discovered, so sdXX's enclosure_device:NN link is removed and one is
added for sdYY. And so if I want to figure out which enclosure slot
sdXX is in, I'm in for a hard time.
It would be a simple matter of writing code to allow all the block
devices in a slot to link back to that slot -- we would have to be a
bit more careful of keeping track of what links exist, but it should
be doable.
The wrinkle is that there are also /sys/class/enclosure/ZZZ/NN/device
symlinks that allow going the other way. And it's harder to see how
to express multiple block devices in one enclosure slot.
Thoughts on how to improve our enclosure handling?
IN SAS-2 the SMP DISCOVER response should contain slot information.
Looking at sysfs for my test system:
# lsscsi
[1:0:0:0] disk ATA ST3320620AS 3.AA /dev/sda
[6:0:0:0] disk ATA ST31000528AS CC38 /dev/sdb
[6:0:1:0] disk SEAGATE ST32000444SS 0006 /dev/sdc
[6:0:2:0] enclosu Intel RES2SV240 0600 -
then fetching the corresponding SAS port addresses:
# lsscsi -t
[1:0:0:0] disk sata: /dev/sda
[6:0:0:0] disk sas:0x5001517e85c3efe5 /dev/sdb
[6:0:1:0] disk sas:0x5000c500215725bd /dev/sdc
[6:0:2:0] enclosu sas:0x5001517e85c3effd -
Referring to the SMP DISCOVER response above, it can be seen
that /dev/sdc is connected to expander phy 7. So getting the
long form output for a SMP DISCOVER on phy 7 :
# smp_discover -p 7 /dev/bsg/expander-6\:0
Discover response:
...
attached SAS address: 0x5000c500215725bd
attached phy identifier: 0
...
device slot number: 255
device slot group number: 255
device slot group output connector:
Sadly the value of 255 means not available. YMMV
My initial thought is that in a multi-path situation, as above, we get
two enclosures appearing as well (one down each path). If we
incorporated the idea of topological subtrees into the identity matching
code, we'd end up filling each of the enclosures with the path connected
devices. That seems to be an easy situation for multi-path drivers to
sort out and one requiring no alteration of the existing enclosure code
(except to do the topological subtree search).
How does that sound?
Does this also solve the problem reported a few weeks back
in which a SES logical unit reported duplicate element
descriptor names?
Doug Gilbert
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
